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Executive Summary 

The Study of School Turnaround (SST) examines the change process in a diverse, purposive sample of 
schools receiving federal School Improvement Grants (SIGs) from 2010-11 to 2012-13. With the 
passage of the American Recovery and Reinvestment Act of 2009 (ARRA), the SIG program underwent 
three major shifts. First, ARRA boosted total SIG funding in fiscal year 2009 to approximately 6.5 times 
the original 2009 appropriation through Title I, section 1003(g) of the Elementary and Secondary 
Education Act (ESEA). As is typically the case with Title I, SIG funds were distributed to states by formula 
based on each state's Title I share; however, states then had to competitively make SIG awards to 
districts with eligible schools. Second, ARRA targeted funds at only the very worst schools— those that 
were in the bottom 5 percent of performance and had been low performing for an extended period of 
time. Third, schools receiving SIG were now required to implement one of four prescriptive intervention 
models believed to be more aggressive and comprehensive than those generally adopted under prior 
policies (Hurlburt, Therriault, & Le Floch, 2012). By increasing the level of funding, better targeting these 
funds to the persistently lowest-achieving schools, and requiring that schools adopt specific intervention 
models, the revamped SIG program aimed to catalyze more aggressive efforts to turn around student 
performance. This report focuses on a small sample of schools receiving SIG during the first year of the 
revamped SIG program (2010-11). 

Study Purpose, Research Questions, and Methodology 

SST is a set of case studies that document the change process during a three-year period in SIG-funded 
schools located in diverse state and local contexts. The case studies are designed to describe the 
characteristics of the schools, the decisions and strategies that the schools and their districts undertake, 
and the challenges they face in attempting to dramatically improve school performance. This report 
presents findings after the first year of funding (2010-11), focusing on the following research questions: 

• How do the contexts of the case study schools differ? How do contexts and stakeholders outside 
the school (e.g., state or district policymakers) influence the adoption and implementation of 
improvement actions in the case study schools? 

• What roles do school leaders play in the improvement process in the case study schools? 

• What specific strategies and actions do the case study schools undertake to improve the 
capacity of teachers and leaders (human capital), the quality of teaching and learning (technical 
core of instruction), and the conditions that support teaching and learning? How are SIG funds 
used to support these strategies and actions? 

• What is the role of SIG in the change process? How do SIG program requirements and the 
supports provided by states and districts contribute to the adoption and implementation of 
improvement actions in the case study schools? 

• Do respondents in the case study schools report that their schools are improving on leading 
indicators (variables that prior research suggests may be related to later student outcomes)? Do 
the case study schools appear to be changing in ways that may foreshadow improved outcomes 
over time? 

Although these questions guided data collection and analyses, SST is exploratory. It does not provide 
definitive answers to these questions, but instead examines and generates hypotheses that might be 
explored in future research. SST does not examine student achievement outcomes and is not designed 
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to document the practices of all, or even necessarily a representative sample of, SIG schools nationwide. 
Rather, SST is an in-depth examination of how SIG funds and strategies are evolving in a small but 
diverse group of SIG schools. 


Box ES.l. Detail on SIG Program 

According to the final rules issued by the U.S. Department of Education (ED) for the SIG program, 
persistently lowest-achieving schools are eligible to receive SIG and include a state's lowest- 
performing 5 percent of schools or five schools, whichever number is greater, in terms of overall 
academic performance for all students, and schools that exhibit a lack of progress toward 
achievement goals. SIG defines three eligibility tiers for persistently lowest-achieving schools, with 
Tier I and Tier II representing the highest priority for SIG funding, and Tier III representing the lowest 
priority. One of four improvement models must be specified for implementation in each Tier I and 
Tier II school identified in a district's SIG application to its state for funding (Hurlburt, Therriault, & 

Le Floch, 2012). The key requirements for each model are as follows: 

• Turnaround model. Replace the principal and no less than 50 percent of the staff, introduce 
significant instructional reforms, increase learning time, and provide the school sufficient 
operational flexibility (e.g., staffing, time, and budgeting) and support (e.g., ongoing, 
intensive technical assistance and related support). 

• Restart model. Reopen the school under the management of a charter school operator, a 
charter management organization (CMO), or an education management organization (must 
enroll, within the grades served, any former student who wants to attend the school). 

• School closure. Close the school and reassign students to higher-achieving schools. 

• Transformation model. Replace the principal, develop a teacher and leader evaluation 
system that takes student progress into account, introduce significant instructional reforms, 
increase learning time, and provide the school sufficient operational flexibility and support. 

These models are consistent with those defined in other ARRA-funded initiatives, including Race to 
the Top and the State Fiscal Stabilization Funds, Phase 2. For more information on SIG 
requirements, see ED's webpage on SIG legislation, regulations, and guidance 
(http://www2.ed.gov/programs/sif/legislation.html). 

The study team collected data from stakeholders at the state, district, and school levels. The school 
sample was selected to include variation in state, district, and school characteristics hypothesized to be 
associated with implementation patterns and turnaround success. Analysts initially identified a base 
sample of 60 schools from the cohort of schools awarded SIG funds in summer 2010. Closure schools 
were not included, and restart schools were oversampled. The final base sample includes turnaround, 
restart, and transformation schools, with the majority of the sample being transformation schools (as it 
is in SIG-funded schools nationwide). From this base sample of 60 schools, we selected three 
subsamples: the core case study sample, the rural sample, and the sample of schools with a high 
proportion of English language learners (ELLs). The 25 core case study schools were the focus of data 
collection in spring 2011 and are the focus of this report. 

The data collection included a teacher survey, fiscal data collection (SIG budgets and audited 
expenditure files), interviews with state SIG personnel, and a site visit from two SST staff members in 
spring 2011 to conduct interviews and focus groups with a range of district and school stakeholders, 
including district officials (i.e., superintendents, SIG directors, and other district personnel), principals, 
teachers, instructional coaches, school improvement teams, external support providers (i.e., 
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curriculum/instructional providers, school turnaround organizations, CMOs), union representatives, 
students (in high schools only), parents, and community members. The complete set of data collection 
instruments can be found at http://www.air.org/topic/education/study-of-school-turnaround-year-one- 
protocol-survey. 

The data from the core case study schools were analyzed by coding transcribed interview notes using 
Atlas.ti® (a qualitative software program) and compiling site visit and survey data into an online data 
repository. The teacher survey data from the core case study schools were then used in conjunction 
with the qualitative data to examine patterns by school level, SIG intervention model, and other school 
characteristics (see Chapter 2 for a more detailed discussion of the sample selection, data collection 
activities, and analytic procedures). 

Throughout this report, we incorporate direct quotations from study respondents. There are two 
primary reasons for the inclusion of quotations: one methodological and the other stylistic. With regard 
to methodology, by providing example quotations with explanations of our analytic measures rather 
than merely describing these measures in the abstract, we can more concretely illustrate how analysts 
coded the raw data (see Appendix B). This approach lends more transparency to how the measures 
were constructed and allows the reader to better judge whether the measures appear well grounded. 
With regard to style, direct quotations enhance the clarity and relevance of the study, which is based 
largely on qualitative data. These data uniquely provide detailed, contextual information that can 
convey meaning through illustrative examples. Quotations were purposefully selected to enrich the 
findings arrived at through systematic, carefully documented analyses. These quotations are not 
representative of all of our data and are only meant to enrich a particular finding, not formally justify it. 

SST Year 1: Key Findings 

Five key findings emerged from the analysis of activities in the core case study schools during the first 
year of SIG: 

• Although all were low-performing, core case study schools differed in their community and 
fiscal contexts, their performance and reform histories, and their interpretations of the causes 
of— and potential solutions for— their performance problems (see Chapter 3). 

• Approaches to leadership varied across the set of core case study schools with most principals 
exhibiting a mix of leadership qualities. The most frequently reported leadership approach 
among the core case study schools was transformational leadership, referring to principals who 
can develop leaders and motivate and engage their staff behind a strong organizational vision 
(see Chapter 4). Although the majority of schools reported some improvement in 2010-11, 
schools in which respondents described the improvements in the greatest number of areas 
also had higher levels of principal strategic leadership (referring to principals who are able to 
formulate a strategy for school improvement and translate that strategy into concrete priorities 
and specific actions) and were more likely to have experienced a disruption from past 
practices (see Chapter 7). 

• For most of the core case study schools, respondents did not perceive SIG as the primary 
impetus for the change strategies that had been adopted. In 19 of these schools, the 
improvement strategies and actions implemented during the first year of the grant (2010-11) 
were reportedly a continuation of activities or plans that predated SIG (see Chapter 5). 
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• At the time of data collection, 7 of the 25 core case study schools had experienced a visible 
disruption from past practice. The remaining schools appeared to be following a more 
incremental approach to improvement (see Chapter 6). 

• Overall, core case study schools with the lowest levels of organizational capacity in 2010-11 
were those in which teachers reported having fewer resources, the SIG award represented a 
larger percentage of the prior year's per-pupil expenditure, and respondents perceived the 
SIG award as a catalyst for change (see Chapter 7). 

Although the Year 1 findings are useful for understanding the context of core case study schools in their 
first year of SIG implementation, these findings are largely preliminary. The trajectory of these schools 
may shift throughout the three years of SIG implementation, and the Year 1 report findings are a first 
step in understanding the complexity of the change process in the core case study sample schools. 
Below, we discuss these key findings in additional detail by reviewing each main chapter in the report. 

Context and Performance Problems in SIG Schools 

A school's context can influence the ways in which stakeholders define the problems they are trying to 
solve and the strategies they use to address those problems. In this report we examine two aspects of 
context: the school's neighborhood context and fiscal context (resources available to the school from non- 
SIG sources). We also examine how school respondents defined the performance problem in their school. 

Analysis of Year 1 site visit data revealed that core case study schools were situated in a range of 
community contexts, from "traumatic" environments (seven schools) to comparatively "benign" 
environments (nine schools). Schools in "traumatic" contexts were located in neighborhoods 
characterized by reports of high crime, incarceration, abuse, and severe urban poverty. In contrast, schools 
in "benign" contexts— although still high-poverty— were characteristically in neighborhoods where limited 
crime was reported, homes were in good repair, and there were few reports of family instability. 

According to respondents, all core case study schools faced challenges with regard to funding and 
resources. In five of the core case study schools, fiscal constraints outside of the SIG award were 
perceived as a barrier to school improvement efforts. Staff at these schools mentioned a variety of fiscal 
challenges, such as staff layoffs, cuts to supplemental programs (such as tutoring, art classes, and field 
trips), increased class sizes, and staff salary cuts. In the remaining 20 core case study schools, fiscal 
constraints were perceived as a challenge but not a barrier to improvement efforts. 

In addition to the role that the school's external context and available resources may play in setting the 
conditions for improvement, SST's conceptual framework posits that the way in which teachers, 
administrators, and parents individually and collectively define their schools' performance problems and 
conceptualize the root causes of those problems will influence how they approach the improvement 
process. The study team organized explanations for schools' histories of low performance into 11 
domains. Among these domains, student behavior was the most commonly reported domain 
(15 schools), followed by the school's internal culture (14 schools), poor instruction/teacher quality 
(12 schools), poor or unstable school leadership (12 schools), the school's external context such as crime 
or poverty (12 schools), lack of engagement from parents/community (11 schools), and teacher 
recruitment or retention (10 schools) (see Exhibit ES.l). 

Respondents in the 25 core case study schools differed in the extent to which they attributed the 
performance problem in their school to factors within their control (internal causes) or outside of 
their control (external causes)— six core case study schools were classified as accepting internal 
responsibility for their performance challenges. In these schools, stakeholders accepted responsibility 
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for their school's challenges by either addressing the challenges or working to improve the school 
despite these challenges. Meanwhile, respondents in five schools attributed their performance 
problems to external factors, such as low levels of parent education or English skills. 


Exhibit ES.l. 

Performance Problems Reported by Core Sample Schools, 2010-11 


Student Behavior 
5chool Culture 
Poor Instruction/Low Teacher Quality 
School Leadership 
School External Context 
Parents/Community 
Teacher Recruitment or Retention 
Challenges with Specific Student Subgroups 
Lack of Student Preparation in Prior Grades 
District Leadership and Support 



20 


Number of Core Sample Schools Reporting 
Performance Problem 


25 


Source: SST respondent interviews and focus groups, spring 2011. 
Note: Includes 25 core sample schools. 


Leadership for Change 

Research and policy suggest that schools engaging in change efforts often have principals who have a 
central role in leading these efforts (Edmonds, 1979; Johnson & Asera, 1999; Picucci, Brownson, Kahlert, 
& Sobel, 2002; Rhim, Kowal, Hassel, & Hassel, 2007; Whiteside, 2006). In this sense, principal leadership 
could potentially be a catalyst for school change, and, if so, a change in the school leader may have 
symbolic as well as substantive purposes in the turnaround process (Herman et al., 2008). SIG guidance 
seems consistent with this hypothesis, as SIG schools adopting either the turnaround or the 
transformation model are required to replace the principal. 

Most schools in the core case study sample (21 of 25) reported replacing their principals in either the 
2009-10 or 2010-11 school year in accordance with SIG guidelines (one school did so twice). Most 
principals in core case study schools (21 of 25) had prior experience serving as principals either at their 
current post or at other schools. They had an average of 5.5 years of experience as principals. Most 
principals (20 of 25) also had prior experience working in low-performing schools. 

When classified on dimensions of leadership (transformational, instructional, and strategic), few 
principals (2 of 25) placed high and few principals (2 of 25) placed low on all three dimensions. According 
to teachers, instructional coaches, and members of the school improvement team, the majority of 
principals (21 of 25) reportedly exhibited a mixture of these qualities. For example, some principals 
received high scores on one or two dimensions of leadership but middling scores on the others. 
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Improvement Actions in SIG Case Study Schools 

The SIG intervention models carry with them requirements and expectations regarding the 
implementation of improvement strategies and actions (see Box ES.l). However, the combined differences 
among the SIG schools— with regard to neighborhood context, fiscal context, ways in which schools define 
the performance problem, and leadership skills, as well as type of SIG model— might lead one to anticipate 
that the improvement strategies and actions adopted by the core case study schools during the first year 
of SIG also would vary. To gain a better understanding of the change processes at each of the core case 
study schools, SST examined the ways in which the 25 schools implemented SIG requirements. 

Respondents at the 25 core case study schools identified 11 improvement strategies and actions 
during the 2010-11 school year. The three improvement actions noted by respondents in the greatest 
number of schools were increasing professional development activities, replacing the principal, and 
increasing learning time. Other strategies included using instructional coaches, replacing teachers, 
changing the core curriculum/instruction, using student-level data, providing student supports, using 
technology, implementing new behavior policies/programs, and providing parent activities. Aside from 
teacher replacement, the implementation of these improvement actions did not appear to vary by SIG 
model. On average, school respondents reported implementing 6 distinct improvement actions per 
school in the first year of SIG, with schools ranging from 3 to 11 improvement actions. Having a greater 
number of reported improvement actions should not, however, necessarily be interpreted as being 
further along in the turnaround process or having greater likelihood of long-term success. 

For 10 of the 11 improvement actions, a subset of study schools reported initiating implementation 
prior to SIG, as shown in dark shading in Exhibit ES.2. In certain cases, SIG funds seemed to supplement 
ongoing improvement efforts by providing funds for previously initiated activities. This is not surprising 
considering that these schools may have been subject to improvement initiatives and accountability 
policies in the past. For example, two schools indicated that teacher replacement procedures had 
occurred prior to SIG. 

Respondents cited a variety of reasons for implementing specific improvement actions. Respondents in 
all of the turnaround and transformation schools that replaced at least 50 percent of their teachers 
during the first year of the grant reported that the schools removed staff who principals perceived to 
be less skilled or motivated. In this way, the schools sought to address a perceived performance 
problem: the need for a more skilled, motivated, and collaborative staff. 
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Exhibit ES.2. 

Number of Schools Adopting Specified School Improvement Actions in 
Core Sample Schools, 2010-11 

Human Capital Management 

Increase professional development 

Replace principal 
Use instructional coach 
Replace teachers 

Technical Core 

Increase learning time 

Change core curriculum/instruction 
Use student-level data 
Provide student supports 
Use technology 

Conditions that Support Learning 

New behavior policies/programs 

Provide parent activities 

Number of Schools 

■ Initiated Prior to SIG Initiated Concurrent with SIG 


Source: SST respondent interviews and focus groups, spring 2011. 

Note: Includes 25 core sample schools. 

Meanwhile, rather than an identified school need, SIG requirements appeared to be the main impetus 
for increasing learning time at 14 of the 20 schools that did so for the first time in 2010-11. The six 

remaining schools reportedly increased learning time not only to meet a SIG requirement but also to 
meet the expectations of districtwide reforms with a similar mandate. 

Implementation of the improvement actions may change over time as interventions progress and as 
individuals and organizations interpret the results and modify practices. However, these preliminary 
descriptive findings may be helpful in better understanding the ways in which core case study schools 
interpreted and implemented SIG requirements during the first year of SIG implementation. The findings 
also serve as a baseline for exploring any changes that may emerge in later years. 

SIG and the Change Process 

Describing the improvement strategies and actions is useful for understanding the initial conditions for 
change across the core case study schools, but it does not indicate whether the strategies and actions 
initiated a process that would leverage lasting improvement and alter future performance at these 
schools. The intention of the SIG program is to catalyze dramatic action in low-performing schools, and 
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the SIG program is grounded in a hypothesis that addressing long-standing, intransigent patterns of low 
performance may demand a dynamic, intensive, sustained change process that starts with a disruption 
of what went before (U.S. Department of Education, 2009). SST therefore examined the extent to which 
actions in the core case study schools signaled a disruption from the past. 

Based on findings from the first year of data collection, respondents in 7 of the 25 case study schools 
described a set of activities that together constituted a disruption from the past. Of these schools, six 
underwent a disruption in 2010-11 and one in 2009-10 (the year prior to SIG implementation). The 
reported activities that constituted a disruption from the past included replacing the principal, changing 
the school governance structure, changing the physical plant of the school, or making symbolic changes 
such as renaming the school. 

In an effort to better understand the relationship between perceptions of SIG and prior improvement 
efforts, SST also examined the extent to which school respondents perceived SIG as a catalyst of the 
change process. In 19 of the 25 schools, SIG was not perceived to be the primary impetus for change, 
whether or not the schools had experienced a disruption from the past. In these schools, SIG was 
incorporated into a reform process that had been planned or launched in 2009-10. In four of the 
remaining schools, respondents perceived SIG as the primary impetus for change, while in the last two 
schools the changes were so limited that they could be characterized as business-as-usual. 

Equally important to understanding the role of SIG in the core case study schools is an examination of 
the challenges they faced in implementing the SIG program and the supports they received in their first 
year of implementation. 

Based on findings from the first year of data collection, the process of applying for SIG and delays in 
the receipt of funds posed challenges to early implementation of the grant. Respondents in core case 
study schools varied in their perceptions of the level of involvement of school stakeholders (e.g., 
principal, school improvement team, instructional coaches, parent representatives) with regard to the 
SIG application process. In 10 of the schools, school stakeholder involvement in the SIG application 
process was limited and in 6 schools there was no stakeholder involvement. Delayed funding was most 
often reported as a constraint on schools' ability to hire new staff, finalize contacts with external 
support providers, and implement plans for extended learning time. 

Most core case study schools (21 of 25) reported receiving at least some support from a state 
education agency, district, or external provider. However, respondents in 20 of the 25 core case study 
schools reported compliance-focused monitoring and guidance, while respondents in 10 core case 
study schools reported receiving support for their improvement efforts. In general, district officials 
reported providing support for improvement more often than school respondents reported receiving 
such support. 

Leading Indicators of Change 

As previously mentioned, SIG was intended to be a substantial intervention for persistently low- 
performing schools, catalyzing dramatic action and yielding quickly perceptible improvements. To 
understand school respondents' assessments of their school's progress (or lack thereof) after the first 
year of SIG implementation, SST analyzed hypothesized leading indicators and explored how perceptions 
of improvement were associated with other school characteristics. 

Respondents in all but 1 of the 25 core case study schools reported at least some initial progress in at 
least one area during the 2010-11 school year (see Exhibit ES.3). The most frequent reports of 
improvement were related to safe and orderly school climate and teacher collaboration. At all 12 
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schools in which respondents reported progress in teacher collaboration and instructional practices, 
respondents also reported actions such as hiring an instructional coach (in the case of improved 
instruction) or increasing professional development (in the case of both improved instruction and 
teacher collaboration). Similarly, among the 14 schools that described an improved school climate, 
respondents in 9 of them also described improvement actions focused on student behavior. 


Exhibit ES.3. 

Number of Core Sample Schools With Perceived Improvement on 
Specified Leading Indicators, 2010-11 


Safe and Orderly Climate 
Teacher Collaboration 
Instructional Practices 
Quality of Leadership 
Use of Data for Instructional Decisions 
Material Resources 
Student Engagement 
Staff Expectations for Students 



20 


25 


Number of Core Sample Schools Reporting Improvement 


Source: SST respondent interviews and focus groups, spring 2011. 
Note: Includes 25 core sample schools. 


The core case study schools in which respondents described improvements in the greatest number of 
areas also had higher levels of principal strategic leadership and were more likely to have experienced 
a disruption from the past. However, reports of improvement— even widespread improvement— do not 
necessarily mean that a school has built the capacity necessary to foster high levels of student 
achievement. Prior research has identified a number of specific variables, or school conditions, that are 
often present in schools with higher than expected student achievement (Bryk, Sebring, Allensworth, 
Luppescu, & Easton, 2010). We explored the status of each case study school with regard to a set of 
eight indicators of capacity: leadership, coherence, clear and shared goals, teacher collaboration, 
teacher-teacher trust, safe and orderly climate, use of data to inform instruction, and the extent to 
which respondents described an internal or external responsibility for performance problems. 

Overall, core case study schools with higher capacity in the previously mentioned domains also 
reported greater access to material resources. Conversely, core case study schools with lower capacity 
were those in which teachers reported having fewer material resources. These lower capacity schools 
also tended to receive SIG awards worth a larger percentage of what was spent overall in 2009-10 per 
pupil and perceived the SIG awards as a catalyst for change. 
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Conclusion 

The data for the first year illustrated the diversity among core case study SIG schools in terms of 
neighborhood context, fiscal context, and principal leadership, as well as in the ways in which 
respondents defined the performance problem. The core sample schools engaged in a wide range of 
improvement actions and uses of SIG funds, although not all schools had a large infusion of funds and 
the SIG models did not necessarily dictate what actions the schools initiated. That the case study schools 
were using SIG funds to implement different change strategies could be expected given the diversity 
across schools and the fact that many had been the subject of improvement initiatives and 
accountability policies over the years. The schools were thus not blank slates for reformers to craft 
anew. Rather, they were existing organizations with prior reform histories in which the participants tried 
to leverage change by addressing identified performance problems as well as implementing specific 
requirements of the SIG program. 

Given these variations, it is not surprising that, at the end of the first year of SIG implementation, some 
schools appeared to be better positioned to improve student outcomes. However, these findings only 
represent an initial glimpse into the change process at these core case study schools. Subsequent years 
may reveal additional complexities: Schools that started strong in their first year of implementation may 
lose ground in upcoming years, and schools that fared poorly in the initial process may show 
improvement as time passes. Ensuing reports will continue to track the progress of these schools, their 
improvement efforts, and the role of the SIG program. 
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Chapter 1: Introduction 


The Study of School Turnaround (SST) is examining the school improvement process in a diverse, 
purposive sample of schools receiving School Improvement Grants (SIGs) under Title I, Section 1003(g) 
of the Elementary and Secondary Education Act (ESEA) from 2010-11 to 2012-13. The SIG program, first 
authorized in 2001, provides formula-based federal funds to states that then competitively award these 
funds to districts applying for SIG on behalf of their low-performing schools. These schools use the funds 
to implement reforms intended to turn themselves around. SIG funding was increased about 6.5 times, 
and the design and requirements of the SIG award revamped, with the passage of the American 
Recovery and Reinvestment Act of 2009 (ARRA). These modifications were designed to better target SIG 
to the nation's lowest-achieving schools and to ensure that more aggressive improvement strategies are 
adopted than had been previously. Since the passage of ARRA, four cohorts of schools have received SIG 
as of the 2013-14 school year. Cohort I grantees include schools that received SIG during the fiscal year 
(FY) 2009 competition cycle to implement reforms beginning in the 2010-11 school year. Cohort II 
grantees include schools that received SIG during the FY 2010 competition cycle to implement reforms 
beginning in the 2011-12 school year. Cohorts III and IV were awarded for the 2012-13 and 2013-14 
school years. This report focuses on the actions of a purposive sample of Cohort I SIG recipients in the 
first year of implementation during the 2010-11 school year. 

Policy Overview 

Congress introduced provisions to ESEA in 1988 to hold schools accountable for improving the 
performance of their students. The 1994 ESEA authorization (Improving America's Schools Act) tied 
these provisions to state-adopted standards in reading and mathematics, and introduced the notion of 
adequate yearly progress (AYP). It was not until 2001, however, when Congress passed the No Child Left 
Behind (NCLB) Act, that ESEA incorporated national criteria to identify low-performing schools and 
delineated a set of required actions and interventions intended to improve student outcomes in schools 
that failed to meet AYP targets. By 2008-09, 12,599 schools nationwide had been identified for 
improvement, corrective action, or restructuring under Title I of ESEA (Taylor, Stecher, O'Day, Naftel, & 
Le Floch, 2010). Of these, 5,017 schools were in restructuring status, meaning that they had failed to 
meet AYP performance targets for at least five years (Taylor et al., 2010). 

New SIG provisions, bolstered by a substantial infusion of ARRA funds, seek to strengthen the program 
in two ways. First, SIG provisions serve to reinforce the federal government's prioritization of the 
lowest-achieving schools by ensuring that resources are allocated to those schools most in need. 
Although NCLB requirements aim to identify low-performing schools, the AYP criteria do not focus only 
on schools with the lowest overall performance. For example, an NCLB-identified school may have only 
missed AYP targets for one subgroup or a single subject area, rather than for all subgroups and both 
English language arts (ELA) and mathematics. Second, the revamped SIG program aims to catalyze more 
aggressive efforts to turn around student performance by requiring that schools adopt one of four 
specific intervention models and (with ARRA funds) providing greater resources to do so. Although NCLB 
delineates a set of corrective actions, identified schools tend not to adopt the most aggressive 
approaches for turnaround. For example, schools that failed to meet AYP targets for at least five years 
(and were thus in restructuring status) have five options: replace all or most of the school staff, allow 
the state to take over the school, reopen the school as a public charter school, contract with a private 
entity to manage the school, or implement "any other major restructuring of the school's governance 
arrangement that makes fundamental reforms, such as significant changes in the school's staffing and 
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governance, to improve student academic achievement in the school and that has substantial promise 
of enabling the school to make AYP as defined in the State plan" (No Child Left Behind [NCLB], 2003). 
However, the National Longitudinal Study of NCLB found that in 2006-07, only 22 percent of the schools 
in the restructuring (implementation) stage had put in place one of the first four (the more stringent) of 
these options (Taylor et al., 2010). Similarly, an earlier U.S. Government Accountability Office (GAO) 
report found that about 40 percent of the schools in restructuring had not implemented any of the five 
restructuring options in the law (U.S. GAO, 2007). 

The SIG Program Under ARRA 

Authorized under Title I, Section 1003(g) of ESEA and supplemented and amended by ARRA, the SIG 
program targeted more than $5 billion during. FY 2009-2012 to the nation's persistently lowest-achieving 
schools to be used during a three-year implementation period (2010-11 to 2012-13 for Cohort I and 
2011-12 to 2013-14 for Cohort II) (see Exhibit 1.1). 1 Each state's allotment of SIG funds is determined by a 
formula based on Title I allocations. State education agencies (SEAs) then competitively award funds to 
local education agencies (LEAs) with eligible schools. According to U.S. Department of Education (ED) 
guidelines, states may award LEAs up to $2 million annually for each qualified SIG school. 2 States may 
award SIG funds to LEAs and to schools that meet the criteria established by the federal guidelines and in 
accordance with state determinations of LEA capacity and commitment to support school turnaround. 
Between 2009 (when ARRA took effect) and 2011, SEAs held competitions for two cohorts of LEAs. Cohort 
I, the focus of this study, includes districts and schools that received SIG funds to implement reforms 
beginning in the 2010-11 school year. 

The final rules issued by ED in November 2010 defined both the criteria for selecting eligible schools and 
the authorized intervention models. To encourage states to target the lowest-achieving schools, eligible 
schools are defined as belonging to one of the following three tiers: 

Tier I includes any Title I school in improvement, corrective action, or restructuring that (1) is among the 
lowest-achieving 5 percent of the schools in these categories in the state or (2) is a high school that has 
had a graduation rate below 60 percent for a number of years. States have the option of identifying Title 
l-eligible 3 elementary schools that (1) are no higher achieving than the highest-achieving school in Tier I 
and (2) have not made AYP for at least two consecutive years or are in the state's lowest quintile based 
on proficiency rates. 

Tier II includes any secondary school that is eligible for but does not receive Title I, Part A, funds and (1) 
is among the lowest-achieving 5 percent of such secondary schools in the state or (2) has had a 
graduation rate below 60 percent for a number of years. States also may identify Title l-eligible 
secondary schools that (1) are no higher achieving than the highest-achieving school identified as a 
persistently lowest-achieving school in Tier II or have had a graduation rate of less than 60 percent for a 
number of years, and (2) have not made AYP for at least two consecutive years or are in the state's 
lowest quintile based on proficiency rates. 


1 For more information on SIG, including FY 2013-14 funding and regulations for a third and fourth cohort of SIG 
grantees, see the U.S. Department of Education's webpage on SIG legislation, regulation, and guidance 
(http://www2.ed.gov/programs/sif/legislation.html). 

2 The Consolidated Appropriations Act (2010) raised the maximum funding amount for a participating school from 
$500,000 to $2,000,000 per year. 

3 Title l-eligible schools refer to those schools that do not receive Title I funds but may meet the criteria for 
obtaining the funds. 
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Tier III includes the remaining Title I schools in improvement, corrective action, or restructuring that are 
not Tier I schools. States have the option of identifying as Tier III schools Title l-eligible schools that (1) do 
not meet the requirements to be in Tier I or Tier II, and (2) have not made AYP for at least two consecutive 
years or are in the state's lowest quintile based on proficiency rates. 

According to the federal guidelines, SIG funds may be awarded to LEAs to support Tier III schools 
implementing improvement strategies; however, Tier I and II schools must be served first. 


Exhibit 1.1. 

Annual Federal Appropriations for SIG, 2007-2012 


Fiscal Year 

Amount 

Funding Recipients 

2007 

$125,000,000 

Pre-ARRA grantees 

2008 

$491,265 

Pre-ARRA grantees 

2009 

$3,546,000,000* 

Cohort 1 grantees: 

Years 1,2, and 3 of implementation (2010-11 to 2012-13) 



Cohort II grantees: 

2010 

$546,000,000 

Year 1 of implementation (2011-12) 

2011 

$535,000,000 

Year 2 of implementation (2012-13) 

2012 

$535,000,000 

Year 3 of implementation (2013-14) 


Source: U.S. Department of Education School Improvement Grants website: http://www2.ed.gov/programs/sif/funding.html. 
Originally published in Hurlburt, Therriault, & Le Floch (2012). 

Notes: Each grantee school typically receives an award to implement reforms for three years. States with fiscal year (FY) 2009 
carryover funds (i.e., unused funds from their Cohort I competition) were allowed to use these funds to make similar three-year 
awards in their Cohort II competition. Thus, Cohort II grantees also include schools awarded SIG through carryover funds from 
FY 2009. 

■"Includes the regular appropriation of $546 million from Title I, Section 1003(g), as well as $3 billion from ARRA. 

To encourage school districts and schools to adopt aggressive turnaround strategies, an LEA must 
specify one of four improvement models to be implemented for each Tier I and Tier II school identified 
in an LEA's SIG subgrant application (Tier III schools were not required to implement one of the four 
models). These models are consistent with those defined in other ARRA-funded initiatives, including 
Race to the Top and the State Fiscal Stabilization Fund— Phase Two. 4 The key requirements for each 
model are as follows: 

1. Turnaround model: Replace the principal and no less than 50 percent of the staff, introduce 
significant instructional reforms, increase learning time, and provide the school sufficient 
operational flexibility (e.g., staffing, time, and budgeting) and support (e.g., ongoing, intensive 
technical assistance and related support). 

2. Restart model: Reopen the school under the management of a charter school operator, a charter 
management organization, or an education management organization (must enroll, within the 
grades served, any former student who wants to attend the school). 

3. School closure: Close the school and reassign students to higher-achieving schools. 


4 For more information on Race to the Top and the State Fiscal Stabilization Fund, see the U.S. Department of 
Education's webpages on these initiatives: http://www2.ed.gov/programs/racetothetop/index.html 
http://www2.ed.gov/programs/statestabilization/index.html. 
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4. Transformation model: Replace the principal, develop a teacher- and leader-evaluation system 
that takes student progress into account, introduce significant instructional reforms, increase 
learning time, and provide the school sufficient operational flexibility and support. 

Box 1.1 summarizes the demographic characteristics, models, and funding levels of the initial cohort of 
SIG schools nationwide. 


Box 1.1. Key Findings From the Cohort I Baseline Report 

SIG-awarded schools. Among the 49 states (and the District of Columbia) with available data, 1,228 
schools were awarded SIG funds. Consistent with the program's intent, SIG-awarded schools were 
more likely to be high poverty (68 percent of students in SIG schools were eligible for free or 
reduced-price lunch compared with 45 percent of students nationwide). They also were more likely 
to be high minority (73 percent of students in SIG schools were non-White compared with 45 
percent of students nationwide), located in urban areas (53 percent of SIG schools were in large or 
middle-sized cities compared with 26 percent of schools nationwide), and high schools (40 percent 
of SIG schools were high schools compared with 21 percent nationwide). 

Intervention models. The transformation model was adopted for nearly three fourths (74 percent) 
of SIG-awarded Tier I and Tier II schools. In 16 states, the transformation model was the only 
intervention model adopted for SIG-awarded Tier I and II schools. The turnaround model was 
adopted for 20 percent of SIG-awarded Tier I and Tier II schools, whereas the restart and school 
closure models represented 4 percent and 2 percent, respectively, of SIG-awarded Tier I and II 
schools. 

Total SIG awards. School-level SIG amounts varied by tier and state. The average total award among 
Tier I and Tier II schools was $2.54 million compared with $520,000 among Tier III schools. The 
average three-year award for Tier I and Tier II schools varied across states, from $620,000 in 
Vermont to $4.63 million in Illinois. High schools received the largest average total allocation ($2.37 
million), whereas elementary schools received, on average, $1.37 million. 

Relative size of SIG awards. The relative funding levels among SIG schools varied across states. In 
four states, Tier I and Tier II SIG funds were worth 6 percent or less of what was spent overall in 
2009-10 per pupil. (The average 2009-10 spending in these states ranged from $10,700 to $13,400 
per pupil.) In 11 states, Tier I and Tier II SIG funds were worth 30 percent or more of what was spent 
overall per pupil in 2009-10. (The average 2009-10 spending in these states ranged from $6,400 to 
$23,500 per pupil.) 

Source: Hurlburt, Le Floch, Therriault, & Cole (2011). 


Study Purpose and Conceptual Approach 

SST describes the change process in a subset of Cohort I SIG-funded schools. SST is designed to describe 
the characteristics of the schools, the decisions and strategies that the schools and their districts 
undertake, and the challenges they face as they attempt to dramatically improve school performance. 
During a period of three years beginning in the 2010-11 school year, SST followed the case study schools, 
which are situated in a variety of state and local contexts, and documented what happens in these schools. 
SST does not examine student achievement outcomes and is not designed to provide a snapshot of the 
practices of all, or even necessarily a representative sample of, SIG grantees nationwide. Rather, SST is an 
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in-depth examination of how SIG funds and strategies are evolving in a variety of participating schools. This 
is the first report from the study, which covers the first year of SIG implementation. 5 

Research Questions and Conceptual Framework 

SST examines the change process in the study's sample of SIG-funded schools. Specifically, SST describes 
the improvement strategies and actions that case study schools adopt and implement, reasons that key 
stakeholders have for undertaking these strategies and actions, and changes that take place over time in 
the functioning of the schools and the strategies they employ. This report seeks to set the stage by 
focusing on the following broad set of research questions: 

1. How do the contexts of the case study schools differ? How do contexts and stakeholders outside 
the school (e.g., state or district policymakers) influence the adoption and implementation of 
improvement actions in the case study schools? 

2. What roles do school leaders play in the improvement process in the case study schools? 

3. What specific strategies and actions do the case study schools undertake to improve the 
capacity of teachers and leaders (human capital), the quality of teaching and learning (technical 
core of instruction), and the conditions that support teaching and learning? How are SIG funds 
used to support these strategies and actions? 

4. What is the role of SIG in the change process? How do SIG program requirements and the 
supports provided by states and districts contribute to the adoption and implementation of 
improvement actions in the case study schools? 

5. Do respondents in the case study schools report that their schools are improving on leading 
indicators (variables that may be related to later student outcomes)? Do the case study schools 
appear to be changing in ways that may foreshadow improved outcomes over time? 

Although these questions guided data collection and analyses, SST is exploratory. It does not provide 
definitive answers to these questions but instead examines and generates hypotheses that might be 
explored in future research. 

The study addresses the research questions using a conceptual framework based on the SIG program 
requirements and the research literature on organizational change processes, policy implementation, 
and effective schools (see Exhibit 1.2). There is a vast research literature on improving low-performing 
schools. Numerous studies of these schools have hypothesized relationships among a range of 
programmatic and organizational variables and improvements in teaching and learning. SST's conceptual 
framework reflects the assumptions and hypotheses that many researchers, policymakers, and 
practitioners hold about the factors that foster school improvement. The framework is used to identify 
and define the main constructs, guide the development of data collection instruments, and focus the 
analysis of data. 


5 In addition to this report, SST will produce a final report and briefs on two special topics: SIG schools with a high 
proportion of English language learners and rural SIG schools. 
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Exhibit 1.2. Conceptual Framework 
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Contextual Influences (all levels): past reform efforts; fiscal resources; 
implementation of other federal programs (Title I, II, III; Race to the Top); 
other relevant state and local policies; stakeholder relations and political 
climate; external school environment (e.g., safety, community support) [Ch3] 

State Implementation of SIG 

State SIG policies: SIG eligibility, selection process for school districts, 
support and guidance 

Dimensions of implementation: coherence, specificity, authority 


District Implementation of SIG 
Definition(s) of the performance problem: needs assessment, staff 
interpretation of performance data, staff interpretation of SIG, goal 
setting 

Strategies and actions in key improvement domains (descriptive): 

human capital management (e.g., teacher evaluation system, 
professional learning, principal replacement, support); technical core 
(e.g., curriculum and instruction, data use); financial resource allocation 
Dimensions of implementation: consistency, divergence from prior 
practice, specificity, authority, sustainability 

Improvement Activity in SIG Schools 
Definition(s) of the performance problem: needs assessment, 
staff interpretation of performance data, staff interpretation of 
SIG, goal setting [Ch3] 

Leadership for change: transformational, instructional, strategic, 
distributed [Ch4] 

Strategies and actions in key improvement domains [Ch5]: 

• Human capital management: staff recruitment and 
replacement, professional learning, evaluation and support 

• Technical core: curriculum and pedagogy, use of time, data 
use, student supports 

• Conditions to support teaching and learning: parent and 
community, school climate, financial resource allocation 

Dimensions of implementation: consistency, divergence from 
prior practice, breadth, depth, buy-in, sustainability 


External 
Partners 
Key improvement 
domains: 

professional 
learning, student 
supports, use of 
time, parent and 
community 
Dimensions of 
implementation: 
consistency, fit, 
intensity, stability, 
accountability 


Leading Indicators of School Improvement OverTime [Ch71 

Collaboration and collective responsibility, safe and orderly school 
climate, trust among staff, clear and shared goals, quality of 
leadership, culture of continuous improvement (relying on data), 
coherence, staff stability, staff expectations of students, student and 
teacher engagement 

♦ i 



Student outcomes 


Student outcomes in 


Student outcomes in 

before SIG 


Year 1 


Years 2-3 
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The evidence supporting the relationships depicted in Exhibit 1.2 is mixed, and some of the relationships 
have stronger empirical support than others. A complete review of the evidence base for all of the 
hypothesized relationships is beyond the scope of this study. We use the research literature primarily to 
identify the constructs and relationships that may contribute to the change process in low-performing 
schools, according to researchers, policymakers, and practitioners. 

Assumptions. Undergirding the framework and design are several assumptions drawn from 
organizational theory and the literature on school change: 

• The primary focus of improvement efforts lies within the school— its people, activities, resources, 
and relationships (O'Day, 2002). The change process can occur in any one or a combination of 
areas within a school (e.g., strategies to improve human resources, curriculum and instruction, 
parent and community involvement, and school climate) (Axelrod & Cohen, 1999). 

• At the same time, school performance is influenced by the systems in which these schools are 
situated (O'Day & Bitter, 2003; Smith & O'Day, 1991). Thus, SST is designed to examine systemic 
contributors to chronic low performance and systemic approaches to addressing this low 
performance. The strategies that states, districts, and schools select and employ reflect different 
theories of action, including varying conceptions of the problems to be addressed and different 
assumptions about how the chosen strategies will address those problems (Argyris & Schon, 
1974; City, Elmore, Fiarman, & Teitel, 2009; Weiss, 1995). SST seeks to understand what people 
do to improve outcomes and processes in the lowest-performing schools and their reasons for 
doing so. 

• The quality of implementation mediates the effect of any policy or program; therefore, 
implementation takes shape as policies and practices are interpreted and acted on across 
multiple levels of the system (McLaughlin, 1987; Spillane, Reiser, & Reimer, 2002). 
Implementation also may change over time as interventions progress and as individuals and 
organizations interpret the results and modify practices. 

• Improvement actions are comprised both of what is done and how (or how well) it is done. For 
example, regarding activities associated with professional development, one could measure the 
characteristics of the professional development activities (what types of professional 
development activities occurred, over what period of time, and for which staff) or various 
analytic dimensions of the implementation process (alignment with the instructional program or 
other aspects of human capital management, such as teacher evaluation, intensity and duration, 
level of targeting, teacher buy-in to the professional development approach or activity, and 
sustainability over time). This study will measure both the descriptive characteristics of the 
change strategies and the analytic dimensions associated with their implementation. 

• Schools are complex social systems (Honig, 2006). The characteristics of schools and the various 
improvement strategies they employ interact and overlap. They are situated in a wide variety of 
district and state contexts. Such complexity makes attribution of causality difficult if not 
impossible, especially from limited case data. SST's objective, therefore, is to describe and 
analyze how school-level actors interpret the performance problems of their schools, 
approaches they take to address these problems, and conditions schools face, rather than to 
predict or assess effects. 

Reflecting these assumptions, each box in the framework provides a description of the constructs that 
this study examines, paying special attention to those that were given most attention in the first year of 
data collection, which is the subject of this report. The framework also depicts the hypothesized 
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relationships among aspects of school improvement, represented by the arrows in the framework. 
(Chapter references provide the location of topics covered in this report.) Because the school is the 
central focus for the SIG program and this study, the discussion starts there. 

Improvement Activity in SIG Schools 

The box labeled Improvement Activity in SIG Schools (see Exhibit 1.2) is the core focus of this study: to 
document over time the change process in a purposive sample of SIG schools attempting to improve 
their persistently-low performance. Within this box are four aspects of the change process in these 
schools: the actors' conceptions of the performance problem, the aspects of leadership that may 
catalyze and guide the change process, the specific strategies and actions that schools select and 
implement, and the key analytic dimensions or qualities associated with implementing these actions and 
strategies. In examining improvement activity in SIG schools, this report focuses primarily on the 
definition of the problem, leadership, and selected improvement strategies (the first three aspects) 
because the schools were still in the initial phase of implementing SIG activities during the first year of 
data collection (2010-11). In the second and third years of data collection, the study will explore the 
implementation of the improvement strategies in greater depth, as well as the evolution of the school 
change process over time. 

Actors' definitions of the performance problem. Some researchers have posited that individual and 
collective definitions of the performance problem, implicit and explicit theories of causality and change, and 
interpretations of strategies and their apparent results can shape the improvement strategies and actions of 
the school actors (as well as the district and support providers) (Spillane, Reiser, & Gomez, 2006). 

Leadership approach. According to a synthesis of studies of schools that have rapidly improved student 
achievement, successful turnaround schools appeared to have strong principal leaders who helped 
catalyze change (Herman et al., 2008). Analyses of the principal's role describe a complex set of skills 
among successful principals (Usdan, McCloud, & Podmostko, 2000). The ability to set and clearly 
communicate a vision; knowledge of academic content and pedagogical techniques; knowing how to build 
committed staff and work with teachers to strengthen their skills; understanding how to use student data 
for decision making; and the ability to be informed by as well as rally students, teachers, parents, and 
community partners appear to be among the principal skills associated with rapid school improvement, as 
indicated by case study research (Herman et al., 2008; Usdan, McCloud, & Podmostko, 2000). 

Strategies and actions in key improvement domains. In their attempts to improve student outcomes, 
schools may adopt different strategies and many specific types of actions (which may or may not be part 
of a broader strategy). For the purposes of this study, strategies and actions have been grouped into 
three broad categories or domains: improving the capacity of teachers and leaders in managing their 
performance (human capital management), improving the technical core of instruction (what and how 
students learn), and improving school conditions that support learning. Derived from the SIG guidance 
documents, the study's research questions, and literature on school improvement described below, 

SST's conceptual framework focuses on the following strategies and actions in each broad domain: 

• Human capital management. Human capital management strategies include those aimed at 
ensuring that capable teachers and leaders work in a school (e.g., recruitment, selection, 
placement, and evaluation) as well as strategies to support and motivate school staff (e.g., 
professional learning opportunities and incentives). Theory and correlational research have 
related school success to staff capacity (Aaronson, Barrow, & Sander, 2007; Cohen & Ball, 1999; 
Hanushek, 1986). 
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o Teacher replacement. Low-performing schools are more likely than others to have 
inexperienced, poorly-qualified, or unmotivated teachers (Clotfelter, Glennie, Ladd, & 
Vigdor, 2008; O'Day, 2005), and the emphasis of the SIG program on replacing teachers 
appears to align with correlational evidence that the quality of teachers is related to student 
outcomes (Gordon, Kane, & Staiger, 2006). However, replacing teachers may not necessarily 
result in improved outcomes because it is difficult to predict which teachers are likely to be 
effective in low-performing schools based on characteristics such as experience, education, 
certification, and longevity (Clotfelter, Ladd, & Vigdor, 2006). 

o Professional learning. Staff replacement relies on bringing in individuals with high levels of 
knowledge and skill; however, strategies undertaken to improve the knowledge and skills of 
those already working within the school are more prevalent avenues for increasing capacity. 
Some researchers have posited relationships between the opportunities for professional 
development and collaboration, as well as supportive physical working conditions and 
improved teacher effectiveness (McLaughlin & Talbert, 2006). Researchers also have studied 
the relationships between specific features of professional development and improved 
teacher effectiveness. Professional development activities that are content focused (Garet, 
Porter, Desimone, Birman, & Yoon, 2001; Kennedy, 1998), are consistent with district and 
school improvement goals (Elmore, 1997; Garet et al., 2001), have ample opportunities for 
active learning (Garet et al., 2001), and are carefully designed for curriculum development 
and explicit instruction (Fishman, Marx, Best, &Tal, 2003; Loucks-Horsley, Love, Stiles, 
Mundry, & Hewson, 2009) have been related to teacher effectiveness (Darling-Hammond, 
Wei, Andree, Richardson, & Orphanos, 2009; Desimone, 2009; Penuel, Gallagher, & 

Moorthy, 2011). Respondents in case studies of successful turnaround schools have 
described professional development, tailored to each teacher's needs and involving follow- 
up in the classroom, as a reform strategy (Herman et al., 2008). Nevertheless, few studies 
have rigorously tested the effects of teachers' professional learning on student outcomes 
(Yoon, Duncan, Lee, Scarloss, & Shapley, 2007), and these studies have had disparate results 
(e.g., studies on coaching conducted by Garet et al. [2008, 2010] and Bryk [2010]). 

o Evaluation and performance management. Evaluating teachers and leaders may be another 
vehicle for improving staff capacity; therefore, the SIG program requires schools that use 
the transformation model to develop teacher and leader evaluation systems that take 
student progress into account. Performance information based on student outcomes or 
classroom observations can be used for compensation incentives, decisions about dismissal, 
and individual development. Researchers have begun to study the relationships of these 
uses to teacher and student outcomes (see Cordray, Pion, Brandt, & Molefe [2011], 
Goldhaber & Theobald [2010], and Springer et al. [2010] for examples of recent studies). 

• Technical core. At the heart of most analyses of school improvement is the assumption that 
what occurs in the classroom— that is, what is taught and how— is associated with student 
outcomes (referred to in this report as the technical core of instruction). Building on prior 
research, David Cohen and his colleagues define instruction as the "interaction among teachers 
and students around content, in environments" (Cohen, Raudenbush, & Ball, 2003, p. 122). To 
improve the technical core, schools might intervene in any of the elements of the instructional 
unit (content, students, or teachers), but it is in the interaction of these elements, the authors 
argue, where the key to changing outcomes for students actually rests. Although SST is not 
designed to look at the interaction of these elements per se, it can examine the following 
strategies that schools put in place to influence it: 
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o Curriculum and pedagogy. Adopting a new curriculum and teaching strategies can be 
associated with increased student achievement; the What Works Clearinghouse 6 provides 
examples of specific mathematics, science, and literacy curricula that produce achievement 
gains. Although SST is not collecting observation data on classroom instruction, the study is 
collecting interview data on whether schools adopted new curricula or teaching approaches 
as part of their reforms. 

o Data use. Researchers have studied the use of data about student performance and 

outcomes to help teachers fine-tune their practices and catch learning problems before they 
become intractable, possibly resulting in referrals to special education programs (Marston, 
Muyskens, Lau, & Cantor, 2003; McNamara, 1998; Reschly & Starkweather, 1997; Sornson, 
Frost, & Burns, 2005). Researchers also have studied specific strategies to monitor student 
learning and frequent and transparent use of student outcome data to guide instructional 
decisions (Coburn & Beuschel, 2012; Coburn & Turner, 2012a, 2012b). A review of research 
studies found that formative assessments can produce learning gains, especially among low- 
performing students, with "effect sizes larger than those found for most educational 
interventions" (Black & Wiliam, 1998, p.3). Strategies to use data to modify the curriculum 
and teaching appear to be a common feature of turnaround schools (Herman et al., 2008). 

o Learning time. Researchers have long studied increasing the time for learning, either 
throughout the school day, before and after school, or over the course of the school year 
(see Mass 2020, "Time and Learning, a Brief Review of the Research" or a compilation of the 
many studies). A National Research Council synthesis of research on learning explains that 
learning is most likely to endure when students have the opportunity to encounter subject 
material through a mixture of learning contexts and media; such variety is more likely to 
occur when the time is available to engage in several separate but related and mutually- 
reinforcing activities (Bransford, Brown, & Cocking, 2000). SIG program requirements 
emphasize the adoption of strategies to increase time for learning, and this study explores 
whether and how SIG schools adopt such strategies. 

o Student supports. Researchers have studied a variety of academic and nonacademic 

supports, hypothesizing that they can improve student learning. For example, several meta- 
analytic reviews of the research have found one-on-one tutoring to be effective in raising 
student achievement (Chappell, Nunnery, Pribesh, & Hager, 2011; Lauer et al., 2006; Ritter, 
Barnett, Denny, & Albin, 2009). Some studies have found that nonacademic support, such as 
health and nutrition services, and programs to foster social-emotional learning are related 
to improved academic outcomes (for reviews of the research, see Osher & Kendziora [2010] 
and Osher, Kendziora, Spier, & Garibaldi [2014]). 

• Conditions that support teaching and learning. Finally, schools may adopt strategies and 

actions intended to create conditions that foster teaching and learning. Low-performing schools 
often tackle conditions such as school climate and parent or community support prior to making 
changes in the technical core of instruction (Herman et al., 2008). 

o School climate. Researchers have studied school climates, hypothesizing that a safe and 
orderly school environment and a culture focused on learning are associated with increased 


6 The What Works Clearinghouse is funded by the U.S. Department of Education's Institute of Education Sciences 
and managed under contract by Mathematica Policy Research, Inc. For more information, see 
http://ies.ed.gov/ncee/wwc/. 
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student achievement (Bryk et al., 2010; Datnow, Lasky, Stringfield, & Teddlie, 2006; 
Mosenthal, Lipson, Torncello, Russ, & Mekkelson, 2004; Stringfield & Teddlie, 1991). Case 
study schools that have "beaten the odds" (i.e., whose students achieve at higher-than- 
expected levels) are often perceived as having a safe school environment and a supportive 
climate of mutual trust (Bryk et al., 2010; Herman et al., 2008; Johnson & Asera, 1999; U.S. 
Department of Education, 2010a). 

o Parent and community support. Correlational and case studies have found relationships 
between parent and community involvement and healthy school cultures (Datnow et al., 
2006; Epstein & Dauber, 1991; Rhim, Kowal, Hassel, & Hassel, 2007). Researchers have 
studied how schools work with parents to improve parenting skills, reinforce study skills and 
expectations for their children, and extend learning at home to contribute to student 
learning (Epstein, Coates, Salinas, Sanders, & Simon, 1997). 

o Financial resource allocation. How schools use their resources also may create conditions 
that influence student achievement. The statistical evidence of a relationship between 
spending and student outcomes has been mixed. Although some researchers have 
concluded, based on education production function studies, that there is little evidence to 
support the existence of a relationship between the amount of resources allocated and 
student achievement (Hanushek 1981, 1986, 1989), others have argued that money does 
matter (Ferguson, 1991; Hedges, Laine, & Greenwald, 1994; Murnane, 1991). For example, 
Ferguson and Ladd (1996) found that resources spent to recruit or retain more effective 
teachers and those with more education, along with smaller class sizes, are related to 
student outcomes. Given the temporary nature of the SIG funds, the study focuses on 
whether SIG funds are used in ways that create the conditions for long-term improvement 
efforts and therefore, the sustainability of reforms supported by SIG. 

To summarize, based on the SIG requirements and the literature on school improvement, the study's 
conceptual framework focuses on the strategies and actions that schools undertake in three domains: 
human resources (i.e., the capacities of teachers and leaders), the technical core of instruction (i.e., what is 
taught and how), and the conditions that support instruction. The strategies and actions undertaken in 
these domains may appear to be analytically discrete; however, the study team anticipates overlap 
between actions taken in one domain and those taken in another. Strategies and actions may have 
multiple purposes, and classification into one domain versus another may be somewhat arbitrary. For 
example, a strategy developed to support the professional learning of teachers would likely also be 
intended to improve instruction and contribute to a school climate focused on academic rigor. 
Furthermore, successful schools may combine strategies and actions in different ways. In a recent set of 
case studies, researchers found that schools that had improved student outcomes adopted strategies that 
were well documented in the research literature, but the choice of specific actions and how the strategies 
were combined varied considerably across schools (Aladjem, Birman, Orland, Harr-Robins, Heredia, Parrish 
& Ruffini, 2010). 

Dimensions of implementation. The fourth set of constructs within the Improvement Activity in 
SIG Schools box in Exhibit 1.2 focuses on how, and how well, the strategies and actions are implemented. 
Analytic dimensions of implementation go beyond simply describing the behaviors associated with 
improvement actions to illustrating the strength and qualities of the actions. Many studies of school 
improvement strategies suggest that the success of these strategies depends on the quality of 
implementation (McLaughlin, 2005). Correlational studies indicate that the level and quality of 
implementation are associated with the likelihood of successful turnaround in schools (Newmann & 
Associates, 1996; Newmann, Smith, Allensworth, & Bryk, 2001) and have identified several variables that 
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relate to the implementation of successful improvement efforts. This report examines two of these 
variables: divergence from prior practice and teacher buy-in. 

• Divergence from prior practice may make implementation of a reform more difficult because 
reform efforts, whatever their nature, are attempts to intervene in an ongoing— and sometimes 
entrenched— and complex organizational system. On the other hand, although perhaps more 
challenging to initiate or sustain, efforts that diverge from prior practice may have the best 
prospects for "shaking up" entrenched routines and expectations (see Rhim, Kowal, Hassel, & 
Hassel [2007] and Waters, Marzano, & McNulty [2003]). 

• Teacher buy-in also has been hypothesized to be associated with the successful implementation 
of school reforms (Bailey, 2000; Bodilly, Purnell, Ramsey, & Keith, 1996; Datnow, 2000), 
although it is unclear whether teacher support prior to the reform is critical. Many low- 
performing schools have a long history of failed or abandoned reform efforts, the accumulation 
of which may have dampened staff enthusiasm for undertaking yet another intervention (O'Day, 
2002). Schools historically have attempted to build buy-in by seeking approval from teachers 
and other staff prior to implementing a new school improvement strategy (Bodilly et al., 1996; 
Datnow, 2000). An alternative view is that teachers must begin implementing a reform and see 
the positive effects of the changes before they will "believe" in the reform or "buy in to it." In 
this case, buy-in is a function of the degree to which a practice or a policy is understood and 
accepted by those who are being asked to implement it. 

Leading Indicators 

Achieving desired results in low-performing schools can take some time (Aladjem et al., 2006). Prior to 
observing improved student outcomes, researchers hypothesize that a set of school-level conditions could 
foreshadow student learning outcomes. These conditions could be thought of as intermediate outcomes 
or leading indicators of improvement. The relevant set of conditions identified for this study appears in the 
box labeled Leading Indicators of School Improvement Over Time, at the bottom of Exhibit 1.2. 

Leading indicators— or intermediate outcomes— are components of school climate and functioning that 
in previous studies have been associated with higher-than-expected student achievement. Because of 
these associations, some researchers have also considered such organizational characteristics to be 
indicators of a school's capacity to produce high levels of desired student outcomes (for a brief review, 
see Beaver & Weinbaum [2012]). The study team has identified a set of eight intermediate outcomes 
that are thought to be associated with higher levels of student achievement and that together may 
reflect a school's organizational capacity (see Exhibit 1.3). 

Although there are other potentially-important leading indicators suggested by prior research (e.g., 
changes in teachers' knowledge and skills), the constraints imposed by the study design and resources 
preclude the study team from examining all hypothesized leading indicators. 
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Exhibit 1.3. 

Definitions of Leading Indicators of Improvement 


Leading 

Indicator 

Definition 

Supporting Literature* 

Teacher 

Collaboration 

Teacher collaboration is characterized by 
mutual assistance and support within the 
school context (O'Day, Goertz, & Floden, 
1995). Often described in the literature as 
either same-subject teachers "identifying a 
common curriculum, developing common 
assessments aligned to that curriculum, and 
then analyzing common assessment data to 
make instructional changes" (DuFour, 
2004b) or as teachers of the same students 
but of different subjects working together 
(Erb & Doda, 1989; Rottier, 2001). 

Several studies have found a positive correlation 
between teacher collaboration and student achievement 
(Goddard, Goddard, & Tschannen-Moran, 2007). Teacher 
collaboration and cooperation through the sharing of 
ideas and practices were also found to be associated 
with improved teacher morale and motivation (Corcoran 
& Goertz, 1995). This mutual assistance and support, or 
the "receptivity" of colleagues, was reported by case 
study school respondents to have played a role in 
teachers' daily practice (O'Day, Goertz, & Floden, 1995). 

Safe and 

Orderly 

Climate 

A safe and orderly climate is an 
environment in which students "have a 
sense of being physically and 
psychologically safe in their school. There 
are few disruptions due to disciplinary 
problems, and those that occur are handled 
firmly and fairly" (Consortium on Chicago 
School Research, 2004, Student-Centered 
Learning Climate section). 

A safe school environment characterizes schools that have 
beaten the odds (Bryk et al., 2010; Herman et al., 2008; 
Johnson & Asera, 1999; U.S. Department of Education, 
2010a). "Prevailing research suggests that students' 
feelings of safety at school, and problems with peer 
relationships and bullying, are influenced by a broad array 
of factors, including students' own attributes, attributes of 
their schools, adults with whom students interact, 
families, neighborhoods, and the broader society" 
(Steinberg, Allensworth & Johnson, 2011). 

Teachers' 
Sense of Trust 

Teachers' sense of trust is referred to as the 
extent to which teachers feel they have 
mutual respect for each other, for those 
who lead school improvement efforts, and 
for those who are experts at their craft 
(Consortium on Chicago School Research, 
2004). 

Based on correlational analyses of survey data, Sebring 
and Bryk (2000) found that "in schools that are 
improving, where trust and cooperative adult efforts are 
strong, students report that they feel safe, sense that 
teachers care about them, and experience greater 
academic challenge. In contrast, in schools with flat or 
declining test scores, teachers are more likely to state 
that they do not trust one another" (p.5). 

Clear and 
Shared Goals 

Schools in which there are clear and shared 
goals are characterized by a unity of 
purpose, explicit expectations, and shared 
values for student learning and success 
(Purkey & Smith, 1983; Newmann et al., 
2001). 

Studies of schools with higher-than-expected 
achievement found that establishment of a clearly- 
defined purpose enables a school to "direct its resources 
and shape its functioning toward the realization of those 
goals" (Purkey & Smith, 1983) and helps to reduce 
student alienation (Newmann, 1981). Research about 
organizations other than schools has found that shared 
values among colleagues is related to one's personal 
sense of investment in the organization and facilitates 
cooperation in the workplace (Cable & DeRue, 2002; 
Watrous, Huffman, & Pritchard, 2006). 

Quality of 
Leadership 

A school principal demonstrating quality 
leadership in an improving school is "more 
likely to be an instructional leader, more 
assertive in his/her institutional leadership 
role, more of a disciplinarian, and ... 
assumes responsibility for the evaluation of 
the achievement of basic objectives" 
(Edmonds, 1979). For the purposes of the 
analyses in this report, three dimensions of 
leadership are addressed: transformational 
leadership, instructional leadership, and 
strategic leadership. 

Case studies of successful turnaround schools 
consistently point to the role of the principal in 
turnaround efforts (Edmonds, 1979; Herman et al., 2008; 
Purkey & Smith, 1983). One meta-analysis of 70 studies 
of principal leadership found a positive correlation 
between principal leadership (as measured by teacher 
perceptions) and student achievement (Waters, 

Marzano, & McNulty, 2003). 
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Exhibit 1.3. 

Definitions of Leading Indicators of Improvement (continued from previous page) 


Leading 

Indicator 

Definition 

Supporting Literature* 

Use of Data for 

Instructional 

Decisions 

The use of data for instructional decisions is 
characterized as the monitoring of student 
learning and frequent and transparent use 
of student outcome data to guide 
instructional decisions (Coburn & Beuschel, 
2012; Coburn & Turner, 2012a; Coburn & 
Turner, 2012b). 

Using data to modify curricular and teaching strategies is 
a common feature of turnaround schools (Herman et al., 
2008). Some studies have found that data can help 
teachers fine-tune their practices and catch learning 
problems before they become intractable, in some cases 
diminishing referrals to special education programs 
(Marston et al., 2003; McNamara, 1998; Reschly & 
Starkweather, 1997; Sornson, Frost, & Burns, 2005). 

Programmatic 

Coherence 

Programmatic coherence is measured by the 
degree to which the policies of a school 
reflect consistent goals, the strategies 
employed are clearly designed to foster 
achievement of these goals, and barriers and 
detractors from the goals and strategies are 
systematically removed (Honig & Hatch, 
2004; Newmann et al., 2001). 

Correlational and case studies of schools implementing 
whole-school reforms found that school staff have 
difficulty implementing multiple, unrelated interventions 
(Berends, 2000; Berends, Bodilly, & Kirby, 2002), and 
that isolated interventions not aligned with other school 
or district objectives are less likely to achieve desired 
outcomes than interventions that are closely aligned 
with existing improvement efforts (Datnow et al., 2006). 

Collective 

Responsibility 

Collective responsibility is characterized by 
the way in which school respondents 
attributed the performance problem in 
their school to factors within their control 
(i.e., internal causes) or outside their 
control (i.e., external causes). 

Reviews of research have found that schools in which 
teachers exhibit high levels of collective efficacy and take 
ownership for the challenges facing their schools are 
more likely to improve student outcomes (Bandura, 

1993; Goddard, 2001; Goddard, Hoy, & Hoy, 2000; 
Goddard, Hoy, & Hoy, 2004). 


*The literature referenced in this exhibit includes conceptual as well as empirical work (mainly correlational and case study 
research). The exhibit is not a full review of the research about the leading indicators included in SST's conceptual framework. 
As with the conceptual framework as a whole, the leading indicators reflect the variables that researchers and educators 
hypothesize are related to student outcomes, rather than variables that have been conclusively determined to causally impact 
student outcomes. 


Contextual Influences 

Schools are situated in districts, states, and local communities. Therefore, many factors outside the 
school may shape improvement activity. These factors may include state and district decisions about 
how the SIG program is implemented, and the roles that external partners may play in assisting schools. 
An array of contextual conditions, including reform history, other relevant state and local policies, the 
presence of other federal programs (e.g., Title I, II, or III; Race to the Top), relationships among 
stakeholders (e.g., states and districts with unions), and the political climate can more broadly influence 
what occurs in SIG schools. 

Actors external to the school. The white boxes in Exhibit 1.2 highlight the actions of other system actors 
and the possible influence they may have on what happens in a school. In the case of SIG-funded 
schools, the primary system actors are the district, state, and other external support providers hired to 
facilitate the reform process. 

• State and district implementation of the SIG program. As a federal program, SIG funds flow 
through states to districts and ultimately to schools. In enacting SIG requirements, states and 
districts make decisions that determine which schools receive SIG funds, how much funding they 
receive, and what they can do with those funds. For example, states and districts generally 
determine which models schools will adopt and therefore whether principals and teachers are 
replaced. For these reasons, districts can influence how reforms are implemented at the school 
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level (Berman, McLaughlin, Bass, Pauly, & Zellman, 1977; Elmore & McLaughlin, 1988; Firestone, 
1989; Spillane 1996; Spillane & Thompson, 1997; Sykes, Schneider, & Plank, 2009). In addition to 
variation in approaches, states and districts differ in the nature and amount of support they 
provide to low-performing schools (Taylor et al., 2010). Although states and school districts play 
important roles in SIG implementation, the main focus of SST is on the school, and we explore 
state and district policies largely from the perspective of how they play out in the SIG schools 
that are part of this study. 

• External partners. The SIG guidelines require that funded schools partner in some capacity with 
an external support provider, with the hope that these partners will positively influence the 
adoption and implementation of improvement strategies. There are different types of support 
providers. Educational management organizations and charter management organizations 
provide comprehensive support to schools and have tools and processes that may guide the 
turnaround process. Outside vendors and nonprofit organizations may help a school with one or 
more aspects of the school's improvement strategies (i.e., professional development in 
mathematics or how to collect and manage classroom observation data). Individuals contracted 
with the school district or state (e.g., those affiliated with the statewide system of support) may 
provide direct, long-term assistance. 

Contextual conditions. In addition to the direct influences from other actors in the SIG process (states, 
districts, and external support providers), researchers have examined how the actions that schools take 
also may be related to their reform history; the conditions in the community in which they are located; 
the surrounding political, social, and fiscal environments; and the demographics and needs of the 
students and families they serve. The gray area surrounding the boxes in Exhibit 1.2 represents the 
contextual conditions that can influence SIG implementation at all levels. The dotted borders of the 
school and system boxes in Exhibit 1.2 reflect the permeability of the contextual factors and the 
anticipated influence they will have on the change process in SIG-funded schools. 

In-depth exploration of the full array of contextual influences is beyond the scope of this study. However, 
the study does examine two aspects of context in this report: the school's neighborhood context and the 
fiscal context (i.e., resources available to the school from non-SIG sources). 

• Community context. SIG funds are targeted to low-performing schools, many of which are 
located in low-income neighborhoods. The relationship between neighborhood poverty and 
outcomes for students is well documented. Some studies have found that in comparison to 
children in higher-income communities, children in low-income communities perform more 
poorly in school, have lower skill levels, and more behavioral and health problems, even when 
family characteristics are held constant (Duncan & Raudenbush, 1999; Pebley & Sastry, 2004). 
Some studies also have identified risk factors for poor youth development, including socially- 
disorganized neighborhoods— those characterized by economic and social flux, high resident 
turnover, and a large proportion of single-parent families— as well as neighborhoods where 
crime and drugs are present (Hann & Borek, 2001). Such conditions can be reflected in unsafe 
schools, a particular problem in urban neighborhoods. The incidence of violent episodes— 
including rape, physical attack, and robbery— is almost 60 percent higher in urban than in 
suburban schools and 30 percent higher than in rural schools (Neiman & DeVoe, 2009). 

• Fiscal context. SIG schools are more likely than schools nationwide to have high rates of poverty 
(Hurlburt et al., 2011). Within this group of schools, however, there are substantial differences 
in the state, local, and other federal resources available to them. States and districts differ in 
their overall wealth, tax bases, and levels of funding for education. Furthermore, the economic 
downturn of the last several years has had differential effects on states and localities. How the 
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SIG funds are used may be shaped by the availability of state, local, and other federal resources. 
For example, some schools may receive SIG funds "over and above" a stable state and local tax 
base, while in schools where state and local funding has been cut, SIG funds may largely replace 
prior resources. These differences may shape district decisions about how to allocate SIG funds 
to schools, and district and school decisions about how to use these funds. 

Report Overview 

This report describes the first year of SIG implementation in a purposive sample of SIG schools and 
provides a baseline for subsequent years. In particular, the report describes the study and its sample, 
data collection, and analysis methods, the case study schools and their improvement activity during the 
first year of SIG, and the change process in the case study schools, as follows: 

About the study (Chapters 1-2). Chapter 1 introduces the SIG program, the study, its research 
questions, and conceptual framework. Chapter 2 provides an overview of the study design, the timeline, 
sampling procedures, and the data collection and analytic approaches. 

About the schools and what they are doing (Chapters 3-5). Chapter 3 describes the neighborhood and 
fiscal contexts of the case study schools, and explores how and to what extent these contexts are 
related to respondent perceptions of the schools' performance problems. As principal leadership is both 
a key feature of the SIG models and is associated with school reform in the literature on turnaround, 
Chapter 4 examines the role of principals in the case study schools. Chapter 5 turns to the actions 
undertaken by the schools and the ways in which SIG funds were spent during the first year of SIG, 
including a focused discussion of three specific improvement actions, one in each of the domains in our 
framework— teacher replacement (human capital management), extended learning time (the technical 
core), and programs and policies intended to improve student behavior (conditions to support teaching 
and learning)— to illustrate similarities and differences in approaches that schools took and how they 
explained their rationale for the approaches. 

About the change process (Chapters 6-7). The SIG program aims to turn around the nation's lowest- 
achieving schools. Chapter 6 describes the initial change process in the case study schools, the extent to 
which actions taken during the first year of funding represented a disruption from the past, the role of 
SIG in these actions, and the support provided to assist SIG schools in their improvement activities. 
Chapter 7 describes the condition of the core case study schools in spring 2011, to what extent 
respondents perceived improvement, and where schools stood on potential indicators of future 
outcomes, as reported by school personnel. 

The study explores these aspects of the change process recognizing that 2010-11 was the initial year of 
the revised and expanded SIG program. A major premise of SST is that school improvement is a dynamic 
process — what is observed initially may not presage what comes later. A school that has made little 
progress in the first year may achieve greater focus in later years; a school that seems to be on a positive 
initial trajectory may lose its direction. This report provides a first glimpse of an evolving story that will 
be continued in subsequent years. 
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Chapter 2: Study Overview 

To address the study's research questions, we conducted case studies consisting of site visits that 
included interviews and focus groups with a range of district and school stakeholders, such as district 
officials, principals, teachers, instructional coaches, school improvement teams, external support 
providers (i.e., curriculum/instructional providers, school turnaround organizations, CMOs), union 
representatives, students (in high schools only), parents, and community members. Brief telephone or 
in-person interviews with key district and school personnel (i.e., superintendents, district SIG directors, 
principals, and instructional coaches) supplemented the site visits. Data collection activities also 
included telephone interviews with state officials, teacher surveys, and fiscal document requests (SIG 
budgets and audited expenditure files). This chapter provides an overview of the study design, the 
schedule of data collection activities, our approach to sampling, a description of data collection 
activities, and an overview of analytic techniques. 

Study Design and Timeline 

We collected data from respondents in a purposive sample of SIG schools. The sample was selected to 
include variation in observable state, district, and school characteristics that might be associated with 
implementation patterns and turnaround success. We initially identified a base sample of 60 schools 
from the first cohort of schools awarded SIG funds in the summer of 2010, from which three subsamples 
were selected: a core sample, a sample of schools with a high proportion of English language learners 
(ELLs), and a rural sample. 

The core sample consists of 25 schools, which the study team visited in spring 2011 and spring 2012. In 
spring 2013, the study team visited 12 of these 25 schools. In fall 2011 and fall 2012, the study team 
conducted shorter visits (brief site visits or telephone interviews with only four key respondents). The 
core sample of 25 schools was the subject of the most intense data collections, including interviews and 
focus groups with a range of respondents, a teacher survey, and fiscal data collection. The core sample 
schools are the primary focus of SST (and this report), as they yield the richest information about the 
change process in a selection of SIG-funded schools. 

The ELL sample consists of 11 schools with a high concentration of ELLs, and the rural sample includes 9 
rural schools. These subsamples enable us to explore the change process in two special types of SIG 
schools that are of policy interest. The core, ELL, and rural samples are not mutually exclusive: 5 of the 
11 ELL sample schools are also in the core sample, and 4 of the 9 rural sample schools are also in the 
core sample. One school is in both the rural and ELL samples, and one school is in all three samples. For 
the ELL sample, data were collected through a site visit, including interviews and focus groups with a 
range of respondents in fall 2011 and fall 2012. Analogous site visits were conducted at the rural sample 
schools, but only in spring 2012. 

Exhibit 2.1 summarizes the data collection activities in each of the three SST years. 
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Exhibit 2 . 1 . 

Main Study Components and Schedule of Data Collection Activities 



Base Sample: 60 schools 

State Interviews: spring 2011, 2012, and 2013 



Core Sample: 25 Schools 

- Site visits: spring 2011 and 2012; 12 schools visited in spring 2013 

- Brief visits (telephone or in-person interviews with key staff): fall 
2011 and 2012 


-Teacher survey: spring 2011, fall 2011,and fall 2012 
- Teacher survey supplement: spring 2012 and 2013 


ELL Special Topic Case Studies: 

Rural Special Topic Case Studies: 

11 schools (5 overlap with core) 

9 schools (4 overlap with core) 

- Site visits: fall 2011 and 2012 

; - Site visits: spring 2012 

- Teacher survey: spring 2011, fall 

- Teacher survey: spring 2011 and 

2011, and fall 2012 

j fall 2011 

- Teacher survey supplement: fall 

j - Teacher survey supplement: fall 

2011 and 2012 

/ 2011 


Sample Selection 

The sampling plan was designed to support an examination of SIG-funded schools in a variety of 
contexts. The plan involved selecting case study schools that varied across a number of dimensions, 
described in detail below. The sample is not a nationally representative sample of SIG-funded schools 
and is not intended to provide findings that are necessarily representative of SIG schools nationwide. 
Rather, the intent was to design a sample that would enable us to collect in-depth data on the change 
process in a variety of persistently low-performing schools. We sought variation on observable state, 
district, and school characteristics that might be associated with implementation patterns. At the school 
level, the following factors were taken into account in selecting the sample: 

• Tier I and Tier II schools. This study focuses on Tier I and Tier II schools, as they are the only 
schools required to implement one of the four SIG intervention models under SIG regulations 
(U.S. Department of Education, 2010b). They are also the schools that SIG designated as being 
the highest priority for funding. 

• School intervention model. Since our unit of analysis is the school, closure schools were 
excluded from the sample because those schools would not be available for longitudinal study. 
Restart schools were oversampled to ensure a sufficient number in the sample to study. 
Therefore, our sample includes turnaround, restart, and transformation schools, with the 
majority of the sample being transformation schools, as is the case in SIG-funded schools 
nationwide. 
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• Grade level. To reduce the number of dimensions of sample variation to a manageable level in 
light of analytic and resource constraints and to ensure that the dimensions we did select had 
adequate representation in our sample, we defined the base sample to include 30 high schools 
and 30 elementary schools, thus excluding middle schools. These two levels merited inclusion 
for several reasons. Elementary schools were selected, in part, because of the large number of 
low-performing elementary schools. For example, 55 percent of SIG-eligible schools in the first 
cohort were elementary schools, whereas 20 percent were middle schools (Hurlburt et al., 

2011). High schools were selected, in part, because of the challenges faced by students and 
staff. For example, high schools are frequently organized into academic departments, serve 
students with a diverse set of postsecondary goals, and are populated by adolescents who often 
face adult responsibilities (Carnoy, Elmore, & Siskin, 2003; Harvey & Housman, 2004; Le Floch, 
Boyle, Therriault, & Holzman, 2010). High schools also are widely perceived to be the most 
resistant to improvement strategies (and thus merit further inquiry) and are the focus of current 
policy interest, as a convergence of efforts on the part of private foundations, researchers, and 
advocacy groups has focused attention on high schools (Hess, 2005; Hill, 2006; Yohalem, 
Wilson-Ahlstrom, Ferber, & Gaines, 2006). 7 Although middle schools face their own unique set 
of issues that deserve attention, particularly in relation to student behavior, our need to 
minimize the sources of variation outweighed their inclusion in the study. Our findings are thus 
not necessarily relevant to any low-performing middle schools. 

Additional school-level variables considered included school size, school locale or urbanicity, 
demographics of enrolled students (i.e., percentage eligible for free or reduced-price lunch, percentage 
minority, and percentage ELL), and the size of the SIG award as a percentage of overall annual per-pupil 
spending. 8 

At the state level, we sought to ensure variation on a small set of key variables, including: 

• Union policies (i.e., right-to-work versus unionized) 

• Level of 2009-10 per-pupil spending on education 

• Level of SIG funding per school 

• Region 

Finally, we sought to select districts with multiple SIG-funded schools— in part to facilitate analyses of 
the district role in the change process and in part to limit data collection costs. 

Base sample. From the universe of Cohort I SIG-funded schools in 49 states and the District of 
Columbia, 9 we identified 60 schools— 30 elementary and 30 high— in 24 districts from 6 states. To 
generate a purposive sample of schools, districts, and states, analysts used a two-step sampling design. 


7 Initiatives such as the Bill and Melinda Gates Foundation's Small Schools Initiative and Early College High School 
Initiative and Achiever's American Diploma Project, the Center for Research on the Education of Students Placed At 
Risk at Johns Hopkins University, and the National Governors Association's Honor States are examples of national 
efforts that focus on improving high schools through private foundations, researchers, and advocacy groups. 

8 The percentage for each school was computed as the SIG school's annual SIG per-pupil award as a percentage of 
the per-pupil spending on instruction, support services (student support services, instructional staff, and school 
administration), and operation and maintenance for the year prior to the SIG award, for the district in which the 
school is located. The district measure is a proxy for per-pupil school-level spending (2009-10 base per-pupil 
spending figures from the Common Core of Data [http://nces.ed.gov/ccd/] are CPI-adjusted to 2011 dollars). 

9 At the time the SST sample was selected (March 2011), Hawaii had not announced SIG subgrant awards. 
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First, we selected 6 states from which the 60 base sample schools would be identified. In selecting these 
states, we sought variation in the key state-level dimensions described above. In addition, states were 
required to have: (1) at least five SIG-funded schools meeting the school-level criteria described above 
(i.e., Tier I or Tier II, elementary or high school, and implementing the turnaround, restart, or 
transformation model); and (2) at least one local education agency (LEA) with three or more SIG-funded 
schools meeting these school-level criteria. 

The final sample of schools was selected from the SIG-funded schools in the six selected states, using an 
iterative process through which we sought a balance among urbanicity, SIG models, annual per-pupil 
funding, school size, and student demographics within the constraints of containing the number of 
districts. 

Core sample. Among these 60 base sample schools, we identified 25 schools as the core sample. In 
selecting the core sample, we sought to achieve an approximate balance of intervention models, 
schooling levels, and nesting of case study schools within districts. We also sought to represent all six 
states and urbanicity categories in the base sample. Finally, we aimed to include schools with a range in 
the SIG award sizes relative to overall annual per-pupil spending. The set of 25 schools includes: 

• 13 elementary schools and 12 high schools in 13 districts 

• 16 urban schools, 5 urban fringe schools, and 4 rural schools 

• 13 transformation schools, 9 turnaround schools, and 3 restart schools 

Comparison of Case Study Schools to SIG-Eligible and SIG-Funded Schools 
Nationwide 

Although SST case study schools are not intended to be nationally representative, we sought to ensure 
that the core sample shares some similarities with SIG-funded schools nationwide. Compared to SIG- 
funded Tier I and II schools nationwide (see Exhibit 2.2), the core sample schools are more likely to be 
high-minority, urban schools with larger student enrollments; however, the sample is comparable to 
SIG-funded schools on other variables. (Core sample schools compare analogously to SIG-eligible Tier I 
and II schools nationwide.) Some purposeful distinctions exist between the core sample and SIG-funded 
schools nationwide. Most notably, SST includes only elementary and high schools. With regard to 
intervention model, the core sample features more turnaround and restart schools and fewer 
transformation and closure schools. Although SIG-funded schools nationwide included alternative, 
special education, and vocational schools, the core sample schools only included regular schools. 

Since the unit of analysis is the school, district characteristics were not directly part of the sampling 
criteria. However, because our aim was to nest some of our case study schools within the same districts, 
some differences will necessarily arise between the core sample districts and SIG-funded districts 
nationwide. Most notably, the core sample districts tend to be larger in terms of number of schools and 
average enrollment because most SIG-funded districts nationwide (64 percent) have just one SIG-funded 
school (Hurlburt et al., 2011). 
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Exhibit 2.2. 

Characteristics of Cohort I SIG-Eligible Tier I and Tier II Schools, 
SIG-Funded Tier I and Tier II Schools, and Core Sample Schools 



SIG-Eligible Tier 1 
and Tier II Schools 
(N = 2,141) 

SIG-Funded Tier 1 
and Tier II Schools 
[N = 826) 

Core Sample 
Schools 
(A/ = 25) 

School Level (percentage of schools) 




Elementary 

21% 

24% 

52% 

Middle 

17% 

20% 

0% 

High 

51% 

49% 

48% 

Non-standard 

11% 

7% 

0% 

School Type (percentage of schools) 




Regular 

86% 

91% 

100% 

Alternative 

11% 

7% 

0% 

Special Education 

2% 

1% 

0% 

Vocational 

1% 

1% 

0% 

Charter School Status (percentage of schools) 

12% 

1% 

0% 

Urbanicity (percentage of schools) 




Large or middle-sized city 

54% 

59% 

68% 

Urban fringe and large town 

26% 

23% 

16% 

Small town and rural area 

20% 

18% 

16% 

Free and Reduced-Price Lunch (school average 




percentage of students) 3 

75% 

76% 

81% 

Race/Ethnicity (school average percentage of 




students) 3 




White 

19% 

17% 

8% 

African American 

43% 

46% 

42% 

Hispanic 

32% 

32% 

41% 

Native American 

2% 

2% 

4% 

Asian 

3% 

3% 

4% 

Total School Enrollment (school average) 

614 

676 

831 

SIG Intervention Model (percentage of schools) 




Transformation 


74% 

52% 

Turnaround 


20% 

36% 

Restart 


4% 

12% 

School Closure 


2% 

0% 


Source: 2009-10 Common Core of Data (http://nces.ed.gov/ccd/); state websites. 

Notes: Includes 2,141 Cohort I SIG-eligible Tier I and II schools, 826 Cohort I SIG-funded Tier I and Tier II schools in 49 states and 
Washington, D.C., and 25 core schools in 6 states and 13 districts. 

Percentage values for characteristics with multiple categories may not sum to 100 due to rounding. 

Non-standard refers to those schools with a grade configuration not falling within the elementary, middle, or high school 
categories. 

3 Student characteristics are weighted in proportion to the number of students enrolled in a school. 
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Year 1 Data Collection Activities 

We now describe the spring 2011 data collection activities for SST, which included site visits, interviews 
with state officials, a teacher survey, and fiscal data. The complete set of data collection instruments for 
these activities can be found at http://www.air.org/topic/education/study-of-school-turnaround-year- 
one-protocol-survey. 

Site visits. Site visits took place over two to three days in April, May, and early June of 2011. Two SST 
researchers visited each of the 25 core sample schools and their districts. The lead site visitor conducted 
the interviews, and the second site visitor took notes. During some visits, the second site visitor 
conducted some of the interviews or focus groups, depending on the second visitor's experience and 
comfort level doing so. With the permission of interviewees, conversations were audio-recorded. All but 
four interviews and all of the focus groups were audio-recorded. 

We aimed to interview the following respondents in each core sample school and district: 

• Superintendent or SIG director at the district 

• Principal 

• One or two instructional coaches (e.g., mathematics and English) 

• Four teachers (two mathematics and two English) 

• External support provider 

• Union representative 

We also conducted focus groups with the following respondents: 

• School improvement team 

• Parents and community members 

• Two groups of teachers (core and noncore subjects in high schools) 

• Students (in high schools only) 

Across all 25 schools, the following respondents were interviewed: 

• 3 superintendents 

• 7 district SIG directors 

• 18 other district staff (i.e., assistant superintendents, school turnaround specialists, 
curriculum/instructional support personnel) 

• 27 principals, including 12 principals new to their schools in 2010-11 10 

• 323 teachers (through interviews and focus groups), including 58 teachers new to their schools 
in 2010-11 

• 37 instructional coaches 


10 One school in the core sample is divided into four academies, each with its own principal. Three of these 
principals were interviewed, hence a total of 27 principals. 
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• 22 other school staff (i.e., school administrators, parent/community relations liaisons, support 
personnel) 

• 18 external support providers (i.e., curriculum/instructional providers, school turnaround 
organizations, CMOs) 

• 23 union representatives 

• 103 parents 

• 72 students 

In all 25 schools, we conducted interviews and focus groups as planned, including the principal, 
teachers, instructional coaches, district administrators, and parents. We interviewed external support 
providers in all 14 schools that had any external support providers. On average, teacher focus groups 
included 4.5 teachers, and parent focus groups included 4 parents. Student focus groups, which were 
conducted in all 12 high schools, included an average of 6 students. 

Although the data collection process was generally consistent across schools, there were a few minor 
anomalies. For example, in one school there were initially three individual teacher interviews scheduled 
but due to availability issues those three interviews were combined into one teacher focus group. In two 
schools, staff members (the assistant principal in one and the bilingual coordinator in the other) acted as 
the translator for the parent focus group. It also is important to note that in smaller schools, some 
teachers were individually interviewed and also part of the focus groups. Finally, in six schools, various 
interviews experienced interruptions. 

Teacher respondents were selected in coordination with school personnel (generally the principal or 
other school administrator) to include teachers with different levels of overall teaching experience, 
experience at the current school, grade-level assignments, and subject areas taught (in high schools 
only), with the goal of collecting a variety of perspectives on the schools' history and current change 
strategy. Other respondent groups were fully represented in interviews and focus groups due to their 
small numbers, such as the principal, superintendent, SIG director, union representative, parent liaison, 
instructional coaches, school improvement team members, and external support providers. Still other 
respondent groups, such as parents/community members and high school students, were selected by 
school personnel (generally the principal or other school administrator), with instructions to include as 
diverse a group as was feasible. 

The interviews and focus groups were guided by semistructured protocols that outlined key questions to 
ask and provided critical probes as necessary to ensure that discussion of specific topics of interest were 
consistent across respondents and that respondents were able to describe school improvement 
processes and policies in their own words. To build rapport with school staff, the interview structure 
allowed for conversation and discussion. Interviewers also had the flexibility to follow up on themes that 
emerged during interviews that warranted more attention. Interviewers attempted to both get 
information from interviewees on the topics they were most knowledgeable about and get the 
perspective of all respondents on key issues. 

The interviews and focus groups covered the following topics: 

• Respondent's background 

• Respondent's role and responsibilities in the school 

• School context (demographics, strengths, challenges) and reform history 

• Key improvement actions in place at the school in 2010-11 
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• Role of the district, state, and any external providers in supporting the school 

• How SIG money was being used 

• How other funding sources were supporting school improvement efforts 

• Impressions of SIG at the end of the first year 

Interviews with state officials. The study's conceptual framework posits that school-level turnaround 
processes are shaped by the historical and policy context of each state and its demographic and urban 
characteristics. Interviews with state officials in the six states with core sample schools provided insight 
on state-level decisions with regard to state funding, state contexts (e.g., legal constraints and 
flexibility), and state actions and technical assistance. 

Teacher survey. The primary purpose of the survey was to supplement the qualitative teacher data with 
more representative data from teachers on topics such as instructional leadership, trust, perceived 
coherence, collective efficacy, school commitment, and school resources (see Exhibit 2.7 in this chapter 
for an overview of the survey scales). The survey measures are not measures of change, but rather 
cross-sectional in nature, intended to capture teacher perceptions of the current state of affairs in the 
core sample schools in spring 2011. The study team used items developed for other studies, which 
demonstrated that they could be used to create reliable scales. Most scales were from teacher surveys 
developed by the Chicago Consortium on School Research (CCSR), but others were developed by staff 
from the American Institutes for Research for other national studies of school reform. 11 The survey data 
were used in conjunction with the site visit data to examine patterns by school level, SIG intervention 
model, or other school characteristics. 

From May 10 to August 9, 2011, 10-minute Web-based and hard-copy versions of the survey were 
distributed to all teachers in the core sample schools. Surveys were administered on a rolling basis, once 
district research approval was granted and schools had provided teacher rosters. For schools that 
provided teacher e-mail addresses, initial survey administration and follow-up efforts were conducted 
via e-mail and through postal mail. First, all respondents received an e-mail with a customized link to the 
Web-based survey, which included a brief explanation of the survey and its purpose. Teachers who did 
not respond to the survey within two weeks received weekly e-mail reminders until they responded or 
until survey administration was closed. In addition, nonrespondents received a postcard reminder and a 
hard-copy version of the survey via postal mail with return postage. For schools that did not or could not 
provide teacher e-mail addresses, only hard-copy surveys were administered and follow-up efforts were 
all conducted through postal mail. A total of 1,280 teachers were sent the survey across all 25 schools, 
and 794 returned it for an overall response rate of 62 percent. Each school's response rate varied from 
21 percent to 86 percent. 


For additional information about survey items that were drawn from existing sources, see CCSR's survey 
documentation (https://ccsr.uchicago.edu/surveys/documentation) and the teacher surveys for the National 
Longitudinal Study of NCLB (www.air.org/topic/education/study-of-school-turnaround-teacher-survey-nls-nclb). 
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Exhibit 2.3. Teacher Survey Response Rates, Spring 2011 


Response Rate 

Number of Schools 

75% to 100% 

5 

50% to 74% 

16 

25% to 49% 

3 

0% to 24% 

1 


Source: SST teacher survey, spring 2011. 
Notes: Includes 25 core sample schools. 


Exhibit 2.3 presents the distribution of response rates across schools. Because the primary purpose of 
the survey was to representatively characterize teacher perceptions of activities and conditions within 
each school, we excluded schools with low response rates. Based on a series of exploratory analyses, 
survey results for schools with less than a 50 percent response rate were excluded. The sample used in 
all survey analyses includes all 21 schools with a response rate of at least 50 percent. 12 

Exhibit 2.4 summarizes the school level, intervention model, and school urbanicity characteristics of the 
21 schools in SST's survey analysis sample. 


Exhibit 2.4. 

School Characteristics of Teacher Survey Respondents, Spring 2011 

School Characteristics 

Percent of Teachers 

School Level 


Teaches elementary school 

30.9% 

Teaches high school 

69.1% 

Intervention Model 


Teaches in a restart school 

7.2% 

Teaches in a transformation school 

60.5% 

Teaches in a turnaround school 

32.4% 

Urbanicity 


Teaches in a rural school 

11.7% 

Teaches in a nonrural school 

88.3% 


Source: 2008-09 Common Core of Data, School Improvement Grant. 

Notes: Includes 698 teachers in 21 core sample schools (11 elementary schools and 10 high schools). 


Exhibit 2.5 summarizes— by school level and by intervention model— the characteristics of teachers 
respondents from the 21 schools in SST's survey analysis sample. 


12 Two sets of analyses were conducted to examine whether there was evidence of a nonresponse bias. The first 
examined the relationship between school-level response rate and teachers' responses, while the second 
examined the differences between early and late responders and their responses. Neither analyses pointed 
towards a substantial nonresponse bias. For more information, see Appendix C. 
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Exhibit 2.5. 

Teacher Characteristics of Teacher Survey Respondents, Spring 2011 


By School Level: 

All Schools 

Elementary 

Schools 

High Schools 

Has a Master's degree 

51.0% 

45.6% 

53.3% 

Years of experience 

11.7 

9.8 

12.6 

Years of experience in current school 

5.4 

4.7 

5.8 

Number of classes or sections taught in spring 2011 

4.6 

4.0 

4.9 

Number of students enrolled in classes taught in spring 2011 

62.3 

32.4 

75.7 

Number of teachers 

667-693 

202 - 215 

465 - 478 


By Intervention Model: 

Restart 

Transformation 

Turnaround 

Has a Master's degree 

46.8% 

45.3% 

62.5% 

Years of experience 

3.6 

12.5 

12.1 

Years of experience in current school 

0.9 

6.1 

5.3 

Number of classes or sections taught in spring 2011 

4.7 

4.2 

5.2 

Number of students enrolled in classes taught in spring 2011 

36.2 

62.5 

67.9 

Number of teachers 

47-50 

398 - 420 

216-223 


Source: SST teacher survey, spring 2011. 

Notes: Includes 21 core sample schools (11 elementary schools and 10 high schools). Sample sizes refer to the number of 
teachers used in the analysis. A range is provided when the sample sizes varied across items in the table due to nonresponse. 


Fiscal data. The study team requested complete audited expenditure files— containing object, function, 
resource, and location codes— from each district in the study for the three school years prior to SIG: 
2007-08, 2008-09, and 2009-10. Object codes identify specific expenditure types, such as computers or 
teacher salaries. Function codes classify expenditures by their purpose, such as elementary education or 
facilities management. Resource codes identify the source of funding from which a given expenditure is 
paid, such as Title I, ARRA, or state general fund. Location codes indicate at what building or in what 
department an expenditure occurred. These codes allowed us to estimate expenditures for each of the 
25 core sample schools. The study team requested files containing expenditures from all funding sources 
available to a district, including state and federal categorical funds. A request for documents and files 
(RDF) was sent to each district's chief financial officer or budget director to obtain these files, and files 
were received from all districts in the study, although for two districts, 2007-08 files were unavailable 
due to a change in the districts' computer systems. We also obtained Year 1 SIG budgets from either the 
district's SIG application or, if available, a more detailed or updated budget directly from the district. 
Budgets for 7 schools came from district SIG applications, and budgets for the other 18 schools were 
provided by the district. 

Overview of Analytic Techniques 

Site Visit Analyses 

Interview and focus group notes were transcribed to "near-verbatim" quality by referencing the audio 
recordings from the site visits. These notes were then reviewed by the senior site visitor and revised 
until they met the quality standards established for the study. That is, senior researchers reviewed the 
notes to ensure they were close to a transcription, explained acronyms, and identified the role of 
individuals described in the interviews and focus groups. The qualitative site visit data were then 
analyzed using a five-stage process: 
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• Stage 1: Shortly after site visits, researchers entered descriptive information about site visits 
(e.g., number of completed interviews, data-collection challenges, a description of school 
context) into a preliminary Web-based data repository. 

• Stage 2: The analysis team developed codes based on the preliminary data capture and the 
study's conceptual framework. These codes were pilot tested and refined. 

• Stage 3: All core sample qualitative data (interviews and focus groups) were coded using 
Atlas. ti®. 

• Stage 4: Analysts compiled coded site visit and survey data in a second Web-based data 
repository, synthesizing findings for each school. 

• Stage 5: Analysts conducted cross-case analyses based on the repository data. 

Stage 1: Preliminary data capture. The preliminary data capture was developed using a Web-based 
software program (SurveyGizmo®). Using a Web-based platform allowed site visitors to access the data 
capture while they were still in the field, thus facilitating prompt entry of site visit data. The purpose of 
the preliminary data capture was to systematically record the details of the site visit while they were still 
vivid. This platform did not serve as a primary analytic tool but rather ensured that the site visitors 
communicated key features of the site visit to the study team, highlighted unanticipated issues, and 
noted gaps in data collection that would require follow-up. The preliminary data capture template asked 
site visitors to report information on five topics: (1) site logistics; (2) SIG school characteristics; (3) site 
visit participants; (4) the school environment; and (5) first impressions of school improvement efforts. 

Site visitors were encouraged to complete all preliminary data capture activities while on site and were 
required to finalize the preliminary data capture within two weeks of each site visit. After each site visit 
team completed its preliminary data capture, members of the leadership team reviewed the entire 
entry. If reviewers identified inconsistencies or responses that seemed insufficiently supported by the 
evidence provided or otherwise incomplete, site visitors were required to revise them. Once all 25 
preliminary data captures were complete, analysts studied responses across cases to identify early 
patterns or anomalies that could support the development of a coding scheme (Stage 2). 

Stage 2: Developing and piloting codes. A first draft code list was based on: (1) key components of the 
study's conceptual framework; (2) regulatory requirements of SIG-funded schools; and (3) topics that 
were mentioned by respondents and described in the preliminary data capture. After the overall 
approach to coding was determined and the preliminary code list drafted, codes were piloted with a 
subset of data (specifically, near-verbatim notes from a focus group) to determine whether the set of 
codes covered the topics reflected in the data, whether they were of an appropriate grain size, and 
whether the definitions in the code book were clear. See Appendix A for the final list of codes and their 
definitions. 

Stage 3: Coding. The coding stage was a multistep process that included training, weekly assessments of 
interrater agreement, frequent debriefing, and review of coded data by senior researchers. These 
processes were designed to ensure that study data were coded consistently and reliably. All analysts 
who contributed to coding study data participated in two training sessions. The first was a half-day 
webinar that focused on Atlas.ti®, the qualitative software program used to code SST data. This training 
session covered basic functionality of Atlas.ti® and prepared analysts to code data independently. The 
second training session covered the definitions of all codes (see Appendix A). 

Analysts then coded the near-verbatim notes for every interview and focus group. The unit of coding 
was a segment of text reflecting a given construct. In some cases, this consisted of one or two 
sentences, in other cases, one or two paragraphs. Analysts were trained to capture comparable 
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segments of text for each coded passage, including enough adjacent text to enable the reader to 
understand the data when a coded passage was retrieved from an interview or focus group. The 
following steps were used to assess interrater agreement throughout the coding process: 

• The first of three formal interrater agreement checks was conducted prior to the start of coding, 
and the second and third checks were conducted later in the coding process. For each check, a 
passage of raw data (near-verbatim notes from an interview) was selected and coded 
individually by all analysts. Two senior researchers selected these passages to ensure each check 
included a diverse set of codes, including those that were the most conceptually challenging. 
These two senior researchers leading the coding task coded the passage and created a "key." 
Analysts' work was compared to the key. To participate in the coding, analysts were required to 
match at least 75 percent of their codes to those on the key, and limit any additional codes 
added beyond those in the key to 25 percent of the total number of codes. Analysts who did not 
meet this standard were required to repeat the process with a new passage of data. All 
interrater agreement checks were reviewed individually with analysts to build expertise and 
familiarity with the code list and coding strategy. No analysts were permitted to code site visit 
data until they "passed" an interrater agreement check. Five of eight coders passed the 
interrater agreement checks in three attempts, and the remaining three passed on the fourth 
attempt. The purpose of the two follow-up checks was to ensure that coders were applying the 
codes consistently over time. All coders passed the follow-up checks, with an average interrater 
agreement level of 81 percent. 

• In between formal interrater agreement checks, weekly meetings were held to highlight coding 
challenges and reach consensus on how to proceed. These weekly meetings focused on actual 
data pulled from interviews and focus groups that analysts were finding difficult to code. 
Decisions about necessary changes to code definitions, coding procedures, or the addition of 
new codes were documented by the coding task leader and circulated among team members. 
New codes were rarely added because this process required analysts to revisit coded data and 
apply the new codes. When new codes were added, senior researchers reviewed data files to 
ensure that analysts had reviewed already-coded data and added the new codes where 
appropriate. 

• Ongoing, detailed reviews of coded data were conducted by senior researchers. All data were 
reviewed by senior researchers before being considered ready for the next stage of analysis 
(Stage 4). During each review, the senior researcher would look at all of the coded interview and 
focus group transcripts in a single case. Coded passages would be checked for accuracy and 
consistency both across the individual case and against coding guidelines. If inconsistencies 
were identified, they were communicated to the analyst, who then made the required revisions. 

After all of the interview and focus group data were coded, analysts used Atlas.ti® to run queries that 
helped sort the data (e.g., by code or families of codes). These queries served as the foundation for the 
data repository (Stage 4). 

Stage 4: Data repository. After all data from a case were coded, analysts entered the data in a Web- 
based, password-protected data repository using the SurveyMonkey® platform. The data repository 
consisted of open- and closed-ended questions to summarize the data for each case. The topics 
addressed in the data repository aligned with the conceptual framework and included questions related 
to school context, SIG processes, the perceived performance problem, domains of school improvement 
actions, external support providers, state and district activities, and reports of initial progress. 


28 



Case Studies of Schools Receiving School Improvement Grants 


The data repository served four main purposes. First, the repository was designed to include a full case 
narrative for each school. Second, the data repository ensured data of a comparable level of detail and 
quality across cases, so that analysts could review and compare the synthesized data across schools. 
Third, the data repository allowed analysts to easily view and download data for cross-case analyses. 
Finally, the repository created an "audit trail" 13 since analysts were expected to document the sources 
for the findings synthesized in each question. 

When responding to questions in the data repository, each analyst first ran queries in Atlas.ti®, 
identifying coded text that aligned with each question. The purpose of an Atlas.ti® query is to retrieve all 
the relevant data for a given code from a given case. The coded data consist of excerpts of text which 
the analyst had identified as corresponding to a given code. To facilitate the queries in Atlas.ti®, the 
question stem in the data repository included a list of codes (in brackets) that should be used. For 
example, to respond to the question below, the analyst would retrieve data for a given school that had 
been coded as SIG model selection. 


15. Please describe the rationale for selecting the intervention model for this school. If t Download 
there are divergent reports from different stakeholders, please account for those as well. Please 
be sure to include the perspective of the district administrator.[SIG model selection] 

Before entering data in the repository, the analyst needed to determine if data were provided to 
address the question; that is, whether respondents were given the opportunity to respond to questions 
on this topic. For certain analyses (particularly with regard to principal leadership), analysts determined 
that the qualitative data for some schools were insufficient to support analyses, and these schools were 
excluded from the analysis. 

Next, the analyst reviewed and summarized the coded data for the case school in the data repository. 
The analyst was asked to include information on how many and which types of respondents provided 
data on the case school and to note reasons for which data may not have been reported. If there were 
divergent views within a given school (that is, respondents who expressed conflicting views), the analyst 
documented this as well. The excerpt below (an actual data repository entry for one school) illustrates 
how analysts documented which respondents provided data on a given topic and how many 
respondents expressed consistent views: 

The school 14 principal and district administrator both affirmed that they chose the turnaround 
model because they wanted to replace teachers, and the model gave them the leverage to 
remove 50% of the staff. The principal also claimed that in conversations with parents, "the thing 
that came up was that the teachers weren't good. " He also noted that there had been frequent 
mis-assignments of teachers (e.g., a Kindergarten teacher teaching 7th grade English). In focus 
groups and interviews, teachers, instructional coaches, and other administrators reported that 
they had not been involved in the SIG application process and could not report on the rationale 
for the model selection. 

Closed-ended questions in the repository probed analytic dimensions outlined in the conceptual 
framework (i.e., coherence, divergence from prior practice, buy-in) and were presented as a rubric 


13 An audit trail is documentation that creates a chain of evidence that may be inspected by other researchers who 
seek to reproduce the findings (Lincoln & Guba, 1985). 

14 All descriptions of the core sample schools use pseudonyms, and identifying characteristics of schools and 
individuals have been masked (e.g., the reported gender of respondents was randomly assigned). 
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rating scale. When selecting a rating for the case school, the analyst documented which respondents 
had provided data on a given topic and justified the rating (see Exhibit 2.6). 

Exhibit 2.6. 

Sample of the Online Data Repository 



After the analyst completed the data repository for a case school, the second site visitor reviewed the 
entire entry, ensuring that all responses were sufficiently detailed, documented data sources, provided 
justification for summary statements, and were an accurate depiction of the school. In addition, senior 
researchers reviewed responses across all case schools to ensure that the level of detail was comparable. 

Stage 5: Cross-case analyses. Once the data repository was complete, analysts reviewed the data across all 
schools for a given topic (e.g., initial SIG processes, teacher replacement, leadership). Based on the data 
and the research base associated with each topic, analysts developed classification schemes to categorize 
schools. For example, with regard to the perceived performance problems in each school (described in 
Chapter 3), analysts established decision rules to distinguish between schools in which respondents 
assumed internal responsibility for the history of low performance and schools in which respondents 
focused on challenges external to the school. Analysts extracted the relevant data from the data 
repository, reviewed the evidence for each school, and classified the school accordingly. Across the various 
analyses, analysts used different thresholds for the classifications, based on the substance of the topic as 
well as the data availability. For example, the analysis of perceptions of strategic leadership (described in 
Chapter 4) relied solely on the principal interviews, as other respondent groups were not questioned on 
the topic. Although most analyses in this report are cross-sectional in nature (including for example, the 
teacher survey scale items described in Exhibit 2.7) and not direct measures of change or improvement, 
those analyses that are related to perceived changes from years prior to SIG implementation exclude the 
responses of principals and teachers who were new to their schools in 2010-11. 

After an analyst classified schools based on the defined criteria, a second analyst reviewed the coded 
data and classified the school as well, providing a measure of interrater agreement. Across the school- 
level classifications, the first set of interrater agreement ratings ranged from 72 to 100 percent. In cases 
of disagreement, a senior researcher reviewed the ratings and resolved discrepancies. When the 
classifications were complete, the site lead for each school was required to review the data and the 
rating. When there was a discrepancy between the analyst's rating and that of the site visit lead, the 
study team jointly reviewed the data, returning to the original coded data if necessary. Thus, for every 
school-level classification included in this report, multiple researchers reviewed and approved the 
analysis. For further details on each set of decision rules and definitions of school-level classifications, 
see Appendix B. 
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After the initial school-level classifications were identified, analysts examined associations between 
classifications. For example, were schools in which respondents reported substantial progress in 2010- 
11 also those in a "benign" external context, defined as safe, stable (e.g., low rates of transiency or 
student mobility), close-knit (e.g., neighbors know and provide support to one another), or with higher 
levels of home ownership and building repair than neighboring communities? To conduct these analyses 
(analogous to a cross-tabulation in quantitative analyses), analysts created two-way tables to determine 
if there were any apparent relationships between sets of school-level classifications. 

In summary, all analyses were guided by principles of high-quality qualitative research, including: (1) 
transparent standards of evidence for codes and ratings; (2) documentation of an "audit trail"; (3) 
procedures for verifying consistency of data across cases; and (4) measurement of interrater agreement 
in coding. 

Use of Quotations. Throughout this report, we incorporate direct quotations from respondents, which is 
a standard technique in qualitative case study research (Miles, Huberman, & Saldana, 2013). There are 
two primary reasons for their use, one methodological and one stylistic. 

With regard to methodology, this report includes direct quotations from respondents to lend more 
transparency to the study team's constructed measures and allow the reader to better judge whether 
the measures appear well-grounded in the data. As described above, analysts developed school-level 
categorizations that aggregate qualitative responses from multiple interview and focus group 
respondents (see Appendix B for a detailed description of each measure). To do so, analysts reviewed 
coded data and categorized schools based on documented decision rules, thereby condensing the 
qualitative data into a more systematic and quantitative measure. This conversion process sometimes 
involved making judgments about whether specific quotations from study respondents met the 
established thresholds for a particular categorization. By providing example quotations with 
explanations of how analysts categorized schools rather than merely describing these categorizations in 
the abstract, we more concretely illustrate how analysts applied the decision rules. These quotations 
were not selected randomly but rather to be illustrative of the types of quotations associated with 
particular decision rules so that the reader can more fully understand each categorization. 

With regard to style, quotations enhance the transparency, clarity, and relevance of this study, which is 
based largely on qualitative data. These data uniquely provide detailed, contextual information that can 
convey meaning through illustrative examples. Quotes were purposefully selected to enrich the findings 
arrived at through systematic, carefully-documented analyses. As with the quotations selected for 
methodological reasons, quotations selected for stylistic reasons are not representative of all quotations 
in our data. It is important to bear in mind that these quotations are not used to validate an analysis, or 
to "prove" a particular finding, nor should they be construed to represent the sole evidence on which a 
finding was based. They are only meant to enrich a particular finding by conveying richer contextual 
information that is, by necessity, masked from the study's more systematic aggregate measures. 

Teacher Survey Analyses 

The survey was designed to measure constructs related to contextual influences, selected improvement 
actions, and dimensions of implementation (see Exhibit 1.2). The survey scales include measures of 
principal instructional leadership, principal-teacher trust, school commitment, school resources, shared 
goals, shared values, student behavior, and teacher-teacher trust. Questions from existing surveys were 
used if there was evidence that they provide reliable measures of the target constructs. We assessed the 
quality of the scales by conducting a confirmatory factor analysis on the items separately for each scale 
and by computing the scale reliability (Cronbach's alpha). Information on the eight scales that are the 
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focus of the survey analyses are described in Exhibit 2.7. Scale scores were computed based on the 
mean of the individual items composing each scale. Where one or more item was missing, the scale was 
computed as the mean of the remaining items. 15 The school resources scale and the student behavior 
scale ranged from 1 (major challenge) to 4 (not a challenge). The other six scales ranged from 1 (strongly 
disagree) to 4 (strongly agree). 

We also examined three survey items that were intended to measure the coherence of programs within 
the study schools. These items did not factor together (reliability of just 0.55). Thus, we treat these 
three items, which ranged from 1 (strongly disagree) to 4 (strongly agree), as stand-alone survey items. 
These survey items were abbreviated using the following terms: 

• Program follow-up: Once we start a new program, we follow up to make sure that it's working. 

• Too many programs: I worry that we are adopting too many different programs and practices in 
this school. 

• Alignment with improvement goals: This school generally chooses only those school 
improvement opportunities that fit with our improvement goals and strategies. 


Exhibit 2.7. 

Teacher Survey Scale Items and Scale Reliability 


Teacher Survey Scale Items 

Scale Reliability 

Principal instructional leadership (N = 1,173) 

Reliability = 0.94 

Carefully tracks students' academic progress. 

Understands how children learn. 

Makes clear to the staff his or her expectations for meeting instructional goals. 

Sets high expectations for student learning. 

Actively monitors the quality of teaching in this school. 

Presses teachers to implement what they have learned in professional development. 
Knows what is going on in my classroom. 


Principal-teacher trust ( N = 1,170) 

Reliability = 0.94 

The principal has confidence in the expertise of the teachers. 

1 trust the principal at his or her word. 

The principal takes a personal interest in the professional development of teachers. 

The principal looks out for the personal welfare of the teachers. 

The principal places the needs of children ahead of personal and political interests. 

The principal at this school is an effective manager who makes the school run smoothly. 


School commitment ( N = 1,175) 

Reliability = 0.79 

1 usually look forward to each working day at this school. 

1 wouldn't want to work in any other school. 

1 would recommend this school to parents seeking a place for their child. 



For example, if a teacher answered the first two items listed in the school commitment scale but did not answer 
the third item, his or her school commitment scale value would be the mean value of the first two items only. If 
another teacher skipped the first two items, but answered the third, his or her school commitment scale value 
would be the value of the third item. Across scales, two to four percent of teachers had a missing value on one or 
more scale items. 
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Exhibit 2.7. 

Teacher Survey Scale Items and Scale Reliability (continued from previous page) 


Teacher Survey Scale Items 

Scale Reliability 

School resources (N = 1,168) 

Reliability = 0.73 

Large class size and/or case load. 

Inadequate or substandard facilities. 

Too few textbooks and other instructional materials. 

Textbooks and instructional materials that are not aligned with state standards. 


Shared goals ( N = 1,175) 

Reliability = 0.72 

At this school, we have a common understanding of the objectives we're trying to achieve with students. 
Goals and priorities for this school are clear. 

If teachers in this school work hard, we can meet our school's goals for student achievement. 


Shared values ( N = 1,176) 

Reliability = 0.86 

Most teachers at this school have values and philosophies of education that are similar to my own. 
Most of my colleagues share a focused vision for student learning. 


Student behavior ( N = 1,167) 

Reliability = 0.78 

Poor student discipline. 

Large number of student transfers into this school or your class at various points during the year. 
Low student motivation. 

Low and/or erratic student attendance. 


Teacher-teacher trust ( N = 1,175) 

Reliability = 0.78 

Teachers in this school are comfortable discussing beliefs about teaching and learning. 

Teachers in this school are willing to question one another's views on issues of teaching and learning. 
Teachers in this school trust each other. 



Source: SST teacher survey, spring 2011. 

Notes: Includes 37 schools from the base sample with at least a 50% response rate (20 elementary and 17 high schools). 
Teacher sample sizes are reported in parentheses for each scale. 


Fiscal Analyses 

In 2010-11, we collected audited expenditure files from each district in the study, covering the three 
years prior to SIG awards (2007-08, 2008-09, and 2009-10). These files will help us understand the 
fiscal context of schools and districts around the time the SIG awards were made. Site visitors also 
requested Year 1 SIG budgets from districts for each core sample school. In some cases, districts 
indicated that the best Year 1 SIG budget was the one outlined in the district's SIG application, and this 
was used for budget coding. See Appendix E for the full list of budget codes and their definitions. 

Following each site visit, the site visitors used an online data capture template to enter and code each 
expenditure in the SIG budgets. Each line item was categorized into one of a list of types of expenditures 
developed by the study team as most relevant to the study's research questions. Budget items were also 
coded for the domain(s) to which they applied. 

Expenditure file analyses. We calculated per-pupil operating expenses for each school and district in the 
sample. To make calculations as comparable as possible across different districts and states, we were as 
inclusive as possible in the definition of per-pupil expenditures, excluding only the following categories: 

• Building construction, land acquisition, and other similar capital expenses 

• Community services expenditures (e.g., for public libraries or parent services) 

• Adult education 
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• Debt service 

Some expenses that are often considered capital expenses— such as computer equipment and other 
technology— were purchased by schools using SIG funds. Because SIG expenditures were compared with 
total school expenditures prior to SIG, we concluded that both SIG and pre-SIG expenditures should be 
defined to include the same types of expenditures. So, although these technology purchases are not always 
considered operating expenses in the school finance literature, we did so for the purposes of our inquiry. 

Per-pupil expenditures were calculated for the district as a whole for each district containing a core 
sample school. Per-pupil expenditures were also estimated at the school level, using location codes to 
identify school buildings in the expenditure files. These estimates do not include services provided to a 
school that are accounted for at the district level, which often include special education services, 
employee benefits, transportation, or instructional coaches. 16 

Budget analyses. In addition to computing per-pupil expenditures for each district and school, we also 
coded elements in the SIG budget for each core sample school. Lead site visitors coded each line item in 
the school-level Year 1 SIG budget based on descriptions of each line item in the budget narrative and 
each district's chart of accounts for reference (e.g., teacher stipends, student support staff salaries, and 
technology). Coders also identified the domain(s) from the conceptual framework (e.g., human capital 
management, curriculum and instruction) that identified the expenditure's purpose. 

Using these category and domain codes, expenditures were collapsed for reporting into the following 
categories focusing on the strategy or purpose behind the planned expenditures: 

• Academic student supports (e.g., additional classroom teachers, resource teachers, instructional 
aides, and tutors, including external providers) 

• Nonacademic student supports (e.g., staff or external providers providing support for students' 
social, emotional, and economic needs) 

• Curriculum and/or instructional changes (e.g., classroom materials to support new curriculum or 
instructional strategies) 

• Parent activities (e.g., staff time, materials, or technology for parent involvement or education 
efforts) 

• Professional development (e.g., internal or external training costs, teacher stipends for training 
or collaboration, substitute costs, and materials) 

• Instructional coaches 

• Technology (hardware and software) 

• Extended day (e.g., staff stipends, materials costs, external providers, transportation) 

• Extended week or year (e.g., staff stipends, materials costs, external providers, transportation) 

• Data use (e.g., staff salaries, staff time, technology, and assessments) 

• Strategies to change student behavior and/or increase school safety (e.g., security or detention 
center staff time, materials) 


16 In one case, all staff were accounted for at the district level, so school-level expenditures were very low. In this 
case, average salaries and benefits for each position in the district were calculated and applied to each member of 
the school's staff roster, and school-level expenditures were adjusted accordingly. 
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• Instructional leadership (e.g., administrator salaries, leadership coaching) 

• Teacher incentives (e.g., for recruitment, retention, or evaluation purposes) 

• Early intervention programs (e.g., staff and materials for preschool programs) 

• Indirect costs 

Some expenditures were included in multiple categories. For example, technology that is used to 
communicate more efficiently with parents was categorized under parent activities and technology. 
Similarly, new classroom technology (such as SMART Boards) was categorized under technology and 
curriculum/instruction. 
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Chapter 3: School Context and the Performance 
Problems in SIG Schools 


Although all 25 core sample schools were low performing and had been so for some time, they differed 
in other ways. In particular, the schools were located in different types of external contexts. SST's 
conceptual framework posits that a school's context may influence the ways in which stakeholders in 
(and around) it define the problems they are trying to solve and the strategies (both available and 
selected) they use to address those problems. Researchers have examined a variety of school contextual 
factors, including the socioeconomic conditions of the surrounding community, the demographics of the 
student body, the history of prior reforms in the district and school, state and local policies and 
educational governance structures, and the level and distribution of resources (material, human, and 
institutional) (Boyd, 1992; Oakes, 1989). 

This chapter focuses primarily on the external social and economic challenges with which the schools 
and the students in those schools contend, as well as on the fiscal resources available to the school to 
address the educational implications of those challenges and to meet the overall needs of their 
students. We also present respondents' perceptions of these contextual challenges and their views on 
the relationship between the context and their schools' history of low performance. This chapter is 
entirely descriptive and provides the context for analyses that are described later in this report. For 
example, in Chapter 5 we consider the relationship between perceived performance problems and 
improvement actions, in Chapter 6 we discuss the relationship between SIG models and a visible break 
from the past, and in Chapter 7 we explore the association between a school's external context and 
organizational capacity. 


Box 3.1. Key Chapter 3 Findings 

• Core sample schools were situated in a range of community contexts, from "traumatic" 
environments characterized by reports of high crime, incarceration, abuse, and severe urban 
poverty (seven schools) to comparatively "benign" environments characterized by limited 
reports of crime, homes in good repair, and few reports of family instability (nine schools). 

• Respondents in all core sample schools reported facing challenges with regard to funding and 
resources. In five schools, fiscal constraints were perceived as a barrier to school improvement 
efforts. In seven schools, fiscal constraints were perceived as a moderate challenge. In the 
remaining thirteen schools, perceptions were mixed, as teacher survey data and interview data 
told different stories. 

• Six schools were classified as accepting internal responsibility for performance challenges. In 
these schools, stakeholders accepted responsibility for their school's challenges and sought to 
address these challenges. 


Perceived External Context of Core Sample Schools 

Variation in context was a design feature of this study (see Chapter 2). When selecting the 25 core 
sample schools, the study team sought variation that captured a range of SIG-awarded schools 
nationwide— that is, schools in a range of geographic regions, schools in both urban and rural contexts, 
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schools with larger and smaller SIG awards, schools implementing the three intervention models that 
are the focus of this study, and schools at both the elementary and high school levels. 

In addition to these planned dimensions of contextual variation, the qualitative data revealed 
differences among the schools with regard to the challenges associated with the surrounding 
community. These differences suggested additional dimensions of school context. For example, the 
study team observed that some urban schools were in the heart of a city, in neighborhoods associated 
with a single ethnic group— or sometimes in sections of the city marked by reportedly high levels of 
racial tension— while other urban schools were on the outskirts of rapidly growing, sprawling 
metropolitan areas. Rural schools could be in isolated farming communities or cut off from larger 
communities by an imposing mountain range, both of which might have implications for access to 
instructional and human resources. In addition, high rates of poverty, violence, or crime were not 
completely coincident with urbanicity, region, or other variables used in defining the sampling frame. 
Because poverty, violence, and crime are aspects of a student's environment that may be related to 
learning, we explored whether the core sample schools differed in these sorts of environmental 
challenges (see Box 3.2). 

Core sample schools were situated in a range of community contexts, from "traumatic" 
neighborhoods characterized by reports of high crime, incarceration, abuse, and severe urban poverty 
(seven schools) to comparatively "benign" environments characterized by limited reports of crime, 
homes in good repair, and few reports of family instability (nine schools). In schools in "traumatic" 
contexts, teachers and administrators described extremely challenging circumstances in which students 
witnessed or experienced abuse, had limited stability in their immediate family, and in the cases of 
some immigrant students, had experienced harrowing events in their home countries. As one teacher 
commented, "We end up dealing with what most people would call 'behavior issues.' I wouldn't call a 
lot of them 'behavior issues.' I think a lot of them are psychological trauma issues." Respondents from 
schools in traumatic environments emphasized the influence of those environments on internal 
conditions of the school and its students. For example, such schools reportedly experienced frequent 
lock-downs or were the scene of crimes. 

The following site visitor's account of her visit to one school provides an example of a "traumatic" school 
context: 

This school is situated adjacent to a public housing project in a community that has a long 
history of crime and violence. It enrolls students almost entirely from these projects, and 
administrators and teachers alike explained that many of the students have parents and 
grandparents who also attended the school. The school facilities themselves are in reasonable 
condition, but the entire school is surrounded by a fence and all but one entrance is locked 
throughout the school day to improve security. The surrounding environment can be dangerous. 

The principal commented that the school is "locked down" because of violence in the community 
as often as twice weekly. 
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Box 3.2. Perceived External Context of Core Sample Schools 

The classification of core sample schools by perceived external context is described below (see Exhibit 
B.l for more detail on the analytic procedures). For this analysis, "respondents" refers to the 
preponderance of evidence from all the following respondent groups: district administrators, principals, 
teachers, instructional coaches, students, and parents. 

Traumatic context 

• Respondents reported prevalent crime, dilapidated buildings, unstable families, poor race 
relations (e.g., recent race-related riots), gang activity, high rates of home foreclosures, or high 
rates of unemployment; AND/OR 

• Site visitors observed dilapidated buildings, boarded-up windows on nearby homes, graffiti, 
police activity, broken windows on nearby homes and parked cars, litter (including beer cans 
and/or bottles on school grounds or in the neighborhood), or foreclosure signs in the 
neighborhood. 

Depressed context 

• Respondents reported prevalent crime, unstable families, school or neighborhood disrepair, 
gang activity, high rates of home foreclosures, or high rates of unemployment, but also recent 
improvements in the neighborhood in which the school is situated (e.g., stakeholders reported 
decreased crime in the area, repairs being made to neighborhood parks or school buildings, or 
new housing developments); AND/OR 

• Site visitors observed dilapidated buildings, boarded-up windows on nearby homes, graffiti, 
broken windows on nearby homes or parked cars, litter, or foreclosure signs in the 
neighborhood, but also new housing developments, construction or work crews working in 
nearby parks or buildings or the school building itself. 

Benign context 

• Respondents described the surrounding community as safe, stable (e.g., low rates of transiency 
or student mobility), close-knit (e.g., neighbors know and provide support to one another), or 
with higher levels of home ownership and building repair than neighboring communities; 
AND/OR 

• Site visitors observed green, recently mowed lawns, very little or no litter in the surrounding 
area or on school grounds, relatively new school buildings, or construction or work crews 
working in nearby parks or buildings. 

The students at this school were described by administrators and teachers as high-need, high- 
poverty students who often come from homes where abuse, single-parenting, and incarceration 
were commonplace. While there is a core group of parents who are involved in school activities, 
the majority is not. The principal attributed this lack of involvement to a large concentration of 
young parents who themselves had bad experiences at the school when they were students. She 
also pointed to low academic achievement and a general intimidation and wariness towards 
teachers and administrators. A group of the involved parents tentatively agreed to participate in 
the study's focus group. During this conversation, screaming and foul language were 
commonplace and threats of violence occurred. Parents discussed the importance of teaching 
their students to defend themselves, and one parent said that "the school would be better if the 
teachers could slap the kids around a little"— a comment that the rest of the focus group 
acknowledged as accurate by nodding their heads in agreement. 
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In contrast, schools in "benign" contexts— although still high poverty— were relatively safe and stable. 

As a principal in one such school commented, "This is as safe as any suburban district. If we see graffiti, 
it's unusual." When addressing school improvement challenges, schools in benign contexts were not 
encumbered by the level of psychological trauma that was evident in more challenging schools. 

The following site visitor's account of his visit to one school provides an example of a "benign" school 
context: 

This school is a small school within a large urban district. It serves a community of students and 
families that are predominantly Latino. The school itself is unique within the district because it 
was built in 2005 as part of a large campaign and has only been open to students since 2006. It 
enrolls a much smaller number of students than its neighboring counterparts, a condition that 
makes it extremely popular within its community. Unlike other schools in the sample with more 
traumatic contexts, this school is surrounded by mostly single-family homes that seem fairly 
well-maintained. Parents we interviewed spoke about their close-knit community and their 
support for the school. Students, too, spoke positively about the school. They were glad to be 
attending the school because it was perceived as safer, more academically focused, and more 
supportive than other neighborhood options. There was also a degree of pride of ownership 
about attending a new school with new facilities. While the campus was still gated and fenced 
off, these boundaries were not seen as a strategy to "keep people out" as was the case in some 
of the traumatic schools from the core sample. 

In between these two ends of the continuum were the nine remaining schools whose communities were 
more challenging than those in the benign group, but had enough reportedly stable or improving 
conditions to distinguish them from the traumatic group. Although teachers in such schools noted that 
their students experienced challenging circumstances, the level of severity was less pronounced than in 
the traumatic schools, and they described improving conditions in the immediate neighborhood. For 
example, one such school was situated in a community that had a historically high crime rate, but 
parents and teachers noted that crime rates had been declining. In addition, they pointed to the 
proximity of universities and museums and a new school building. 

In sum, these 25 low-performing schools are situated in a variety of external contexts, and these 
contexts may have implications for a school's performance. For example, other studies have found that 
neighborhood characteristics are associated with educational outcomes for students (Ainsworth, 2002), 
characteristics of children's neighborhoods matter for children's development (Leventhal & Brooks- 
Gunn, 2003), and living in a disadvantaged neighborhood lowers college aspirations among African- 
American adolescents (Stewart, Stewart, & Simons, 2007). Subsequent chapters in this report will 
explore the relationship between external context and features of the change process under SIG. The 
next section discusses the fiscal resources available to schools to address the potential educational 
implications of challenging contexts and meet the overall needs of their students. 

Fiscal Overview of Core Sample Schools 

SST's conceptual framework suggests that the fiscal context of schools (e.g., the resources available to 
schools, fiscal and otherwise) may influence the strategies that school stakeholders use to address the 
schools' performance problems. For example, prior to the implementation of SIG, many school districts 
in the United States experienced substantial funding decreases in 2008 and 2009 as a result of the 
recession and its effect on state budgets. In this context, SIG funds could be perceived as a means 
through which schools could mitigate the impact of a budget shortfall, rather than a substantial 
supplemental grant that could enable a school to turn around a history of low performance. 
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In 4 of the 13 core sample districts, the overall level of per-pupil expenditures declined from 2007-08 
to 2009-10 (the three years prior to SIG) (see Exhibit 3.1). In contrast, three districts spent more in each 
subsequent year from 2007-08 to 2009-10, while in four districts, the level of funding remained relatively 
stable (2007-08 expenditure data were unavailable for the remaining two districts). Changes in 
expenditure levels ranged from a decrease of 18 percent to an increase of 21 percent over the three years 
prior to SIG. 


Exhibit 3.1. 

Overall Per-Pupil Expenditures, by Core Sample District, 2007-08 to 2009-10 
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Source: District-provided audited expenditure files, 2007-08 to 2009-10. 

Notes: Includes 13 districts in the core sample. Districts are listed from highest to lowest in terms of 2009-10 per-pupil 
expenditures. 

(a) District 1 and District 7 were only able to provide expenditure files back to 2008-09. 


The eight schools in districts with declining per-pupil expenditures were located in varying community 
contexts: half were in benign contexts, a quarter were in depressed contexts, and a quarter were in 
traumatic contexts. 

Across these 13 districts, the average proportion of funding from federal sources ranged from 4 percent 
to 24 percent (including ARRA funds) in 2009-10, the year prior to SIG. All core sample districts also 
received funding from other non-SIG ARRA initiatives. On average, ARRA funding (including State Fiscal 
Stabilization Funds and other ARRA initiatives but not SIG) during the 2009-10 school year represented 
five percent of per-pupil expenditures. In 5 of the 13 districts, non-SIG ARRA funds helped offset a 
decline in state funding. That is, funding from other sources (federal, state, and local) declined from 
2008-09 to 2009-10, but ARRA funding more than made up for this cut. 

The size of SIG awards relative to overall per-pupil spending in the year prior to SIG varied across core 
sample schools, with estimated Year 1 per-pupil SIG expenditures worth between 5 percent and 128 
percent of what was spent overall per pupil in 2009-10 (see Exhibit 3.2). Such variation in the size of 
SIG awards relative to overall annual per-pupil spending is also characteristic of SIG-awarded schools 
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nationwide. As reported in Hurlburt et al. (2011), among the 43 states and the District of Columbia with 
available data, annual per-pupil SIG awards for Tier I and II schools in Cohort I, on average, ranged from 
3 percent of overall per-pupil spending in 2008-09 (in Louisiana and Vermont) to 58 percent (in 
Montana). 


Exhibit 3.2. 

Estimated Year 1 SIG Per-Pupil Expenditures as a Percentage of Overall 
Per-Pupil Expenditures in 2009-10, by Core Sample School 
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Source: Year 1 (2010-11) SIG budgets provided by districts or from district SIG applications. 

Notes: Includes 25 core sample schools. All school names are pseudonyms. Overall per-pupil expenditures in 2009-10 are 
shown in parentheses. 


Although SIG funds were intended to bring a large infusion of resources into the lowest-performing 
schools, this variation suggests that the extent to which schools were in a position to leverage SIG funds 
to stimulate changes in practices varied. For two core sample schools, SIG funds were worth double the 
prior year's per-pupil expenditures, but for four schools, SIG funds were worth less than the funding cuts 
they experienced between 2008-09 and 2009-10. 


Perceived Funding and Resource Constraints 

Because several districts in the core sample had experienced budget cuts in the three years prior to SIG, 
and because state and federal budgets were constrained during this time, it is not surprising that 
respondents expressed concerns about the schools' finances. Budget cuts and staff layoffs created an 
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atmosphere of uncertainty in many cases, although in general respondents reported varying degrees of 
challenge due to fiscal constraints. Using site visit and teacher survey data, this section focuses on 
respondents' perceptions of fiscal barriers to change, as well as teachers' reports of school resources. 

For the four schools with unavailable teacher survey data, the school was classified based only on site 
visit data (see Box 3.3). 

Fiscal constraints were perceived as a barrier to school improvement efforts in five core sample 
schools. Respondents at four of these schools mentioned staff layoffs in the past three years and/or 
upcoming layoffs for 2011-12. Respondents also described cuts to professional development (one 
school); cuts to supplemental programs such as tutoring, art classes, vocational programs, or field trips 
(two schools); increased class sizes (two schools); and staff salary cuts (three schools). Respondents at 
individual schools also mentioned shifts from full- to half-day kindergarten, and the loss of summer 
school as a result of recent fiscal constraints. For example, one school leader said, "The school district is 
strapped for resources. They don't have money to provide books, clean the building. ..and teachers that 
are good will leave as soon as they can, trying to get to a magnet school or middle-class neighborhood 
school that has more resources.. .for this school we had to raise about $1.6 million in the first year. We 
had to raise another $1.5 million for the facilities project. Some of it is in startup. We know we have to 
raise cash just to get the kids to an equal starting point." Although SIG funds may have been used for 
some of the materials and personnel who had been cut, teachers still perceived their resources to be 
inadequate, as indicated by survey responses. 

In the remaining 20 core sample schools, the fiscal context was perceived as a moderate challenge 
(seven schools) or perceptions were mixed (13 schools). These schools reported fiscal challenges but 
did not universally indicate that these challenges prevented them from moving forward with reform 
efforts. Still, respondents reported financial difficulties such as shifts from year-long to semester-long 
courses, and the need to choose between two high-priority programs (e.g., staff development or 
student mental health services). Respondents in four schools perceived sufficient resources in some 
areas but not in others. For example, at one school, though a teacher noted upcoming cuts to the parent 
ombudsman and student advisor positions, the principal said the school has adequate district support 
and additional financial support through another federal grant. At another school, resources were 
reported to be generally sufficient, but recent increasing enrollment had created strains on those 
resources. One school principal stated that what the school receives (including the SIG award) is 
adequate for what they need to do, but teacher survey data suggested that teachers nonetheless 
experienced mild to moderate challenges (school resources scale average of 2.77 on a scale of 1 to 4) 
with large class sizes, inadequate facilities, and inadequate or misaligned textbooks and instructional 
materials. 
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Box 3.3. Perceived Funding and Resource Constraints 

The classification of core sample schools by perceived funding and resource constraints is described 
below (see Exhibit B.2 for more detail on the analytic procedures). For this analysis, "respondents" come 
from the following respondent groups: district administrators, principals, teachers, instructional 
coaches, students, and parents. 

Perceived as a barrier to school improvement 

• Qualitative data: At least three respondents mentioned cuts to programs; or at least two 
respondents described cuts to programs and articulated how these had impacted or would 
impact student achievement; or at least two respondents mentioned cuts to programs and 
described "dire" circumstances (e.g., 30 percent of staff being laid off); AND 

• Survey data: School resources scale average was at least 0.5 standard deviations (0.40) below 
the scale mean (2.63). 

Perceived as a moderate challenge 

• Qualitative data: At least one respondent mentioned only minor cuts; AND/OR 

• Survey data: School resources scale average was within 0.5 standard deviations (0.40) of the 
scale mean (2.63). 

Perceptions were mixed 

• Qualitative data: Respondents disagreed (in interview and focus group data) about the 
adequacy of school resources; AND/OR 

• Qualitative data and teacher survey data did not match (e.g., school resources scale average was 
at least 0.5 standard deviations [0.40] below the scale mean [2.63], but qualitative data 
indicated "no fiscal constraints"). 

Not perceived as a challenge 

• Qualitative data: No respondent mentioned funding cuts, fiscal constraints, or resource 
constraints as a challenge for the school; AND 

• Survey data: School resources scale average was at least 0.5 standard deviations (0.40) above 
the scale mean (2.63). 

There were no apparent relationships between actual school resource levels or the per-pupil size of the 
SIG award to a school and the school respondents' perceptions of fiscal barriers. To examine how the 
relationship between resource level and perceived fiscal constraints varied by higher-spending and 
lower-spending schools, schools were ranked in order of estimated per- pupil expenditure in 2010-11. 17 
Schools also were ranked in order of per-pupil Year 1 SIG award and the size of SIG award relative to 
prior year spending. Schools in which respondents reported fiscal barriers were both above and below 
the median in all cases, suggesting no apparent relationship between these measures. 

Defining the Performance Problem 

In addition to the role that the external context and available resources may play in setting the 
conditions for improvement, SST's conceptual framework posits that the way in which school 


17 2010-11 per-pupil expenditures were not available at the time of these analyses, so 2010-11 per-pupil 
expenditures were estimated by adding 2009-10 per-pupil expenditures to 2010-11 per-pupil SIG awards. 
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stakeholders individually and collectively define their school's performance problems and conceptualize 
the root causes of those problems may influence how they approach the improvement process. Such 
conceptualizations may include implicit and explicit theories of causality and change, perceptions of 
external and internal constraints, and interpretations of both strategies and their apparent results. 

Some reformers and researchers emphasize a formal and inclusive needs assessment and planning 
process to define the problem and engage participants in its solution (Duke et al., 2005; Learning Point 
Associates, 2010; Marzano, Waters, & McNulty, 2005). Other researchers and theorists focus on the role 
of implicit beliefs, partial understandings, and interpretations of both strategies and outcomes that 
influence behavior within and outside of formal decision processes (Lipman, 1997; O'Day, 2002; 

Supovitz, 2006; Zuckerman, 2006). 

Overview of Perceived Performance Problems 

Perceived performance problems are defined as those for which at least one of the following 
respondent groups described associated challenges: at least two teachers, the principal, or a district 
administrator (see Exhibit B.3 for more detail on the analytic procedures). We organized the perceived 
performance problems into 11 domains, which are described below and summarized in Exhibit 3.3. 
Among these domains, student behavior was the most commonly-reported domain (15 schools), 
followed closely by the school's internal culture (14 schools), poor instruction/teacher quality 
(12 schools), poor or unstable school leadership (12 schools), the school's external context such as 
poverty or crime (12 schools), lack of engagement from parents/community (11 schools), and teacher 
recruitment or retention (10 schools). Funding was only reported by one school as a reason for its past 
history of poor performance. On average, respondents in the schools identified four factors that 
contributed to the school's persistent low performance, although respondents in one school cited one 
main performance problem while respondents from two schools described as many as seven. 

The 10 domains that were cited in multiple schools capture a wide range of contributing factors: 

Explanations related to students: 

• Student behavior: Respondents at 15 schools identified poor student behavior as a key cause of 
their schools' performance problems. Student behavior could include discipline problems, poor 
student attendance, behavior problems, lack of student motivation, students with emotional 
issues, or weak student engagement. 

• Challenges with specific student subgroups: Respondents at seven schools discussed student 
subgroups as one of the main reasons for their schools' poor history of performance. This 
included mention of specific challenges associated with students with severe mental health 
challenges (e.g., autism, severe emotional disturbance), a high concentration of ELLs, many 
special education students, or a variety of languages other than English spoken in students' 
homes. 

• Lack of student preparation in prior grades: Respondents at five schools identified insufficient 
student preparation as a main reason for the poor history of performance at their school. One of 
these five schools was an elementary school, and the other four were high schools. 
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Exhibit 3.3. 

Performance Problems Reported by Core Sample Schools, 2010-11 


Student Behavior 
School Culture 
Poor Instruction/Low Teacher Quality 
School Leadership 
School External Context 
Parents/Community 
Teacher Recruitment or Retention 
Challenges with Specific Student Subgroups 
Lack of Student Preparation in Prior Grades 
District Leadership and Support 
Inadequate Funding 



Number of Core Sample Schools Reporting 
Performance Problem 


25 


Source: SST respondent interviews and focus groups, spring 2011. 
Notes: Includes 25 core sample schools. 


Explanations related to teachers: 

• Poor instruction/low teacher quality: Respondents at 12 schools identified poor instruction or 
teacher quality as a reason for their school's performance problems. Respondents described 
teachers who did not have the capacity to provide effective instruction, a lack of rigorous 
instruction, or a lack of consistent instructional practices or programs. For example, the district 
administrator for one school noted, "In these low-performing schools, I have observed that a lot 
of teachers warehouse kids and you have worksheets. ..the rigor was not where it needed to 
be." 

• Teacher recruitment or retention: There were 10 schools in which respondents specified 
teacher turnover or problems recruiting teachers as one of the reasons their school had a 
history of poor performance. 

Explanations related to leadership: 

• School leadership: Respondents at 12 schools identified school leadership as one of the reasons 
for their history of poor performance. In these schools, respondents described administrator or 
school leader instability, lack of (or poor) school leadership, or poor communication skills of 
school leaders. 

• District leadership and support: In five schools, respondents described poor leadership by 
district officials, instability at the district level, lack of district support in terms of resources, 
disorganization within the central office, or lack of communication from the district. For 
example, a teacher (who is also the union representative) from one school said, "In the 
timeframe that I've been at this school [nine years], the district has been non-existent. ...The 


45 


Case Studies of Schools Receiving School Improvement Grants 


current [superintendent] I don't think has ever come to the school, and the [superintendent] 
before came once for some special event. It kind of left us out there. I feel like there's very little 
recognition or support for our school." 

Explanations related to the school as a whole or the community: 

• School culture: Respondents at 14 schools identified school culture as a contributing factor to 
their school's history of poor performance. For example, one external provider for a school said, 
"There is an adult culture that gets in the way of putting kids first." The school culture was 
identified as a reason for schools' performance problems if respondents described a culture of 
low expectations, lack of consistency across school personnel in expectations for students, a 
divisive relationship between school leadership and teachers, lack of a clear or shared vision for 
change, staff who are resistant to change, lack of teacher collaboration or teacher isolation, 
poor staff morale, or overburdened teachers. 

• School external context: Respondents at 13 schools identified school external context as one of 
the explanations for their school's poor history of performance. School external context was 
considered if respondents discussed any of the following when asked why they thought the 
school had a history of poor performance: the geographic location, the demographics of the 
students (e.g., concentration of one ethnic group, high poverty, migrant families), a transient or 
unstable community or neighborhood where families move in and out of the neighborhood and 
few in the community have been there for more than a year, neighborhood or school safety, or 
problems related to violence in the neighborhood or school. 

• Parents/community: Respondents at 12 schools described the following challenges associated 
with parents and the community: limited parent engagement or involvement, poor relationship 
with parents and the community, parents not being knowledgeable enough to help their 
children with schoolwork, parents or the broader community not demanding high academic 
achievement or rigor, or lack of value placed on education by parents. 

Attributing the Performance Problem and Taking Responsibility for Addressing 
Challenges 

Overall, respondents in the 25 core sample schools described a range of challenges to which they 
attributed their history of low performance. One aspect of these attributions is the extent to which 
school respondents believe they can exert some control over challenges that are largely external or 
outside of their control. Research suggests that schools in which teachers accept internal responsibility 
and take ownership for the challenges facing their school are more likely to improve student outcomes 
(Goddard, Hoy, & Hoy, 2004). If these schools not only have differing contexts but also differ in the ways 
they perceive these contexts, then we might expect such differences to influence the ways in which they 
address their performance problems. 

We thus examined the degree to which respondents attributed the performance problem in their school 
to factors within their control or outside their control. This analysis drew primarily from interviews and 
focus groups, supplemented by teacher survey data. A key caveat with respect to this distinction is that 
all core sample schools reported facing at least some external challenges, whether from the immediate 
context of the school (e.g., lack of security or isolation), lack of district support, limited financial 
resources, or high levels of student poverty and the associated challenges. It is thus not surprising that 
respondents might describe these challenges to site visitors. The distinction used to identify a school 
that emphasized internal versus external factors is whether the respondents appeared to blame such 
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external conditions for their low performance or whether they describe the context as challenging but 
take responsibility for the performance problems in their school (see Box 3.4). 

Five schools attributed their performance problems mainly to external factors (external 
responsibility). For example, a respondent in one of these schools said, "Parents don't realize the power 
they have. We are lacking empowered parents. It isn't that they aren't empowered. They are just 
apathetic. They have so many other social issues going on. Unfortunately, their child's education isn't a 
priority for them." The adults in this school perceived the school's history of low performance to be the 
responsibility of the parents, and outside of their locus of control. In another one of these schools, 
respondents attributed the school's poor performance history to parents not being able to sufficiently 
help their children with homework. For example, one respondent said that "Low levels of education, 
English language skills, and literacy in their own languages make it difficult for parents to help the 
children with their homework." 

Six schools were classified as accepting internal responsibility for performance challenges. 

In general, respondents in schools classified as having internal responsibility acknowledged external 
challenges but either perceived these outside challenges to be somewhat within their control or 
acknowledged the external factors but described how they could control or improve these performance 
problems. For example, the principal at one school said: 

It's obviously a multifaceted issue where you could refer to a number of socioeconomic factors 
dating back to slavery, classism, the industrial de-evolution. There are a million things to point 
to, but I can't change that the factory in this neighborhood closed. I can't change that there have 
been three murders this school year that have impacted the children in this school. What I can 
change is expectations. I can set out a significantly higher standard for our students and our 
teachers at this school. 

Likewise, a member of the school improvement team at one school cited "things like transfer students 
that come in a month before the end of school" as necessary for the school to address, explaining that 
schools need "to find out how to support them and help them graduate." Respondents in these schools 
acknowledged challenges that could be perceived as outside of their control but nonetheless took 
responsibility for crafting a solution. 

Fourteen schools were in the middle category of limited internal responsibility. A broad spectrum of 
schools fell in this category. Eight were closer to the classification of external responsibility, but they did 
describe one or two challenges that were perceived to be within the locus of control of adults in the 
school. At the same time, one school in this group was closer to the internal responsibility classification, 
but adults in the school described some challenges that were perceived to be outside of their control. 
For example, respondents at one school universally cited students' absenteeism as having contributed 
to the school's poor academic performance because as two teachers put it, "We can't educate them if 
they are not here." Although two teachers suggested ideas for how the school might solve or try to 
decrease absenteeism, teachers generally viewed absenteeism as a challenge outside of their control. 
For example, one teacher at this school said, "We can't teach anyone if they're not in school. Truancy 
law is not being followed. We call home. Someone's got to be responsible for these students not going 
to school. If you want us [teachers] to pick them up, pay us and we'll be a taxi service." However, the 
principal at this school cited poor instruction as one of the reasons for the school's history of poor 
performance. Respondents in these schools thus took limited internal responsibility for the schools' low 
performance by attributing the schools' performance problems to things within their control as well as 
attributing some responsibility to things outside of their control. 
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Box 3.4. Perceptions of Locus of Responsibility for Performance Problems 

School-level classifications on the responsibility for performance problems are as follows (see Exhibit B.4 
for more detail on the analytic procedures).* For this analysis, "respondents" refers to at least two 
teachers or the principal. 

Internal responsibility 

• Qualitative data: Respondents described their performance problems as being within the locus 
of control of the adults in the school; or respondents described the external context as 
challenging but assumed responsibility for the school's history of low performance and did not 
describe these external challenges as insurmountable; or respondents described the external 
context in neutral terms, such as "this school is in a neighborhood with high crime"; AND 

• Survey data: School mean on the collective efficacy survey item was above the overall sample 
mean (3.07) (two schools classified as having internal responsibility did not have adequate 
survey data, so their classification is based solely on the qualitative data). 

Limited internal responsibility 

• Qualitative data: Respondents made statements that attributed the performance problems to 
the external context but also described challenges within the locus of control of the adults in 
the school; or respondents attributed the school's history of low performance to a mix of 
internal and external factors; OR 

• Qualitative data and teacher survey data did not match (i.e., the collective efficacy item mean 
was below the overall sample mean [3.07] but interview and focus group data indicated 
"internal responsibility"; or the collective efficacy item mean was above the overall sample 
mean, but interview and focus group data indicated "external responsibility"). 

External responsibility 

• Qualitative data: Respondents explicitly identified factors external to the school as responsible 
for the performance problem and did not attribute the history of low performance to any 
factors internal to the school (e.g., school culture, instruction, leadership, collaboration); or 
respondents made statements that attributed the performance problems to others outside of 
the school, such as "parent participation is why we are failing. That is the foundation"; AND 

• Survey data: School mean on the collective efficacy survey item was below the overall sample 
mean (3.07) (one school classified as having external responsibility did not have adequate 
survey data, so its classification is based solely on the qualitative data). 

For this analysis, we used a single survey item. Because individual survey items are less precise than survey scales, schools were 
differentiated into just two groups with the survey data: above the mean and below the mean. 

Although respondents at five schools attributed their low performance to factors external to the school, 
none were classified as schools in traumatic contexts. Three of these schools were in depressed 
contexts, and two were in benign contexts. However, two schools in traumatic external contexts took 
internal responsibility, and five took limited internal responsibility for their performance problems. For 
example, one school is located in a traumatic external context, where tensions between racial groups 
have erupted in violence, gang activity is prevalent, and neighborhood safety is a concern. Despite the 
challenges that this external context brings (e.g., spillover of violence into the school), the teachers and 
principal are not only trying to make the school safe but are working to "personalize" the high school 
experience so that they can prevent violence from disrupting the school. For example, the principal at 
this school said, "I need to know that child. In a school of 2,000 it's hard to know all. There are enough 
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adults here to know all 2,000. They should have knowledge of what's going on and then they feed that 
to the counselors. I really believe strongly in personalization." 

Chapter Summary 

The 25 core sample schools, although all low performing, are situated in a variety of contexts with each 
school facing varying sets of potential challenges. For example, core sample schools were situated in a 
range of community settings, from "traumatic" environments characterized by reports of crime, 
unstable families, and severe poverty (seven schools) to comparatively "benign" environments in which 
homes were in good repair, streets were clean, and there were few reports of family instability or 
neighborhood instability (nine schools). All core sample schools reported facing challenges with regard 
to funding and resources, however in just 5 schools were fiscal constraints outside of the SIG award 
perceived as a barrier to school improvement efforts, while in the remaining 20 schools, fiscal 
constraints were either perceived as a moderate challenge (7 schools), or the perceptions of challenges 
were mixed (13 schools). Based on respondents' reports, 6 schools were classified as accepting internal 
responsibility for their schools' performance challenges, while the other 19 schools were classified as 
either taking limited internal responsibility for their challenges (14 schools) or focused on external 
challenges (5 schools). 

With regard to their background and context, the 25 schools defy stereotypes and neat categorizations. 
They are diverse, with each of them exhibiting different profiles across our various contextual measures. 
Exhibit 3.4 illustrates this variation. Although among the 25 schools, Sherbrooke Elementary appears to 
have contextual variables that could be hypothesized to be the most favorable to the school 
improvement process (i.e., four of five green cells and no red cells), most schools reflect a mixed set of 
contextual variables, and in general, there are no clear relationships among the contextual variables. 
Each school's unique set of contextual circumstances may help to shape their change process. The 
implications of these contextual variables for the school change process will be explored later in this 
report. 
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Exhibit 3.4. 

Summary of External Context, Resource Constraints, Locus of Responsibility, Overall 
Expenditures, and SIG Expenditures, by Core Sample School 


School 

Perceptions of 
the External 
Context 

Perceptions of 
Funding and 
Resource 
Constraints 

Locus of 

Responsibility for 
Performance 
Problems 

Estimated 
2010-11 
Per Pupil 
Expenditures 

Estimated Year 1 SIG Per- 
Pupil Expenditures as a 
Percentage of Overall Per- 
Pupil Expenditures in 
2009-10 

Aerovista High 

Depressed 

Mixed 

Limited internal 

$ 5,900 

6% 

Baltimore Bridge Elem. 

Depressed 

Mixed 

Internal 

$ 8,200 

7% 

Big Acorn High 

Depressed 

Mixed 

Limited internal 

$ 9,600 

5% 

Blizzard Bay Elementary 

Traumatic 

Barrier 

Limited internal 

$ 11,300 

7% 

Coral High 

Benign 

Mixed 

External 

$ 9,100 

6% 

Elmsville High 

Benign 

Barrier 

Limited internal 

$ 7,800 

22% 

Gale Secondary 

Benign 

Moderate 

External 

$ 7,700 

18% 

Gillepsie High 

Traumatic 

Barrier 

Internal 

$ 10,100 

10% 

Haven Way Elementary 

Benign 

Moderate 

Limited internal 

$ 5,700 

20% 

Inner Brooks High 

Depressed 

Moderate 

Internal 

$ 14,900 

9% 

Island Bay Elementary 

Traumatic 

Mixed 

Limited internal 

$ 10,800 

120% 

McAlliston High 

Benign 

Moderate 

Internal 

$ 8,100 

32% 

Melon Elementary 

Depressed 

Mixed 

External 

$ 8,300 

7% 

Meribel High 

Benign 

Mixed 

Limited internal 

$ 9,600 

49% 

Paul Bunyan High 

Benign 

Moderate 

Limited internal 

$ 5,900 

16% 

Peregrine Hill Elem. 

Depressed 

Barrier 

External 

$ 13,300 

11% 

Proctor Point High 

Depressed 

Mixed 

Limited internal 

$ 9,400 

13% 

Raven Ridge Elementary 

Traumatic 

Barrier 

Limited internal 

$ 10,600 

23% 

Rossignol Elementary 

Depressed 

Mixed 

Limited internal 

$ 12,500 

16% 

Sawbuck Elementary 

Benign 

Mixed 

Limited internal 

$ 10,300 

20% 

Sherbrooke Elementary 

Benign 

Mixed 

Internal 

$ 13,000 

52% 

Sterling Slope Elem. 

Traumatic 

Mixed 

Internal 

$ 13,400 

6% 

Tyron Elementary 

Traumatic 

Moderate 

Limited internal 

$ 12,600 

128% 

Tyro Trail Elementary 

Traumatic 

Moderate 

Limited internal 

$ 10,800 

8% 

West Marble High 

Depressed 

Mixed 

External 

$ 10,500 

13% 


Source: SST teacher survey, respondent interviews and focus groups, spring 2011. 

Notes: Includes 25 core sample schools. All school names are pseudonyms. 

Cells are shaded to reflect the potential challenge to school improvement: green cells are conditions that may be more 
favorable to school improvement (for example, a benign context or internal responsibility), whereas red cells reflect potentially 
greater challenges (for example, lower per-pupil expenditures or a traumatic context), with yellow cells falling somewhere in 
between. 
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Chapter 4: Leadership for Change 


In the past decade, both empirical studies and theoretical explorations of leadership have suggested 
that leadership in a school setting may come from multiple sources and be distributed across multiple 
individuals and structures (Elmore, 2000; Spillane, Halverson, & Diamond, 2004). At the same time, 
research and policy stress the central role of the principal, especially in leading major change efforts 
(Edmonds, 1979; Rhim, Kowal, Hassel, & Hassel, 2007; Whiteside, 2006). More specific to the goals of 
SIG, case studies of successful turnaround schools consistently point to the role of the principal in 
turnaround efforts (Herman et al., 2008). Principals' contribution to school performance may stem from 
the multiple roles they play in a school organization. Principals have responsibilities for engaging and 
managing their staff, for leading and monitoring their school's instructional program, for allocating 
resources, and for responding to external demands from parents, communities, and policymakers. Each 
of these responsibilities can influence the overall functioning of the school organization. At the same 
time, these different roles may require different skills and can even be in tension with one another. 

Although some scholars have focused on delineating one or more idealized models of leadership (e.g., 
transactional, transformational, instructional, strategic, and so forth [Hallinger, 2003; Lynch, 2012]), 
others have argued that effective leaders draw on a broad array of skills and approaches and apply them 
flexibly as the situation and task demand. For example, some theorists argue that organizations in 
complex and uncertain environments require transformational leaders who can motivate and engage 
their staff behind a strong organizational vision and develop leadership throughout the organization to 
respond proactively and creatively to that uncertainty (Bass & Riggio, 2006). Other theorists have 
posited the need for principals who are instructional leaders, particularly in cases where the 
instructional program in the school is weak (Hallinger, 2003). Instructional leaders are those who are 
knowledgeable about instructional issues and who align activity in the school to improve instructional 
practice to significantly improve student learning (Elmore, 2000). Finally, some theorists stress that 
organizations in need of fundamental change, such as persistently low-performing schools, need 
strategic leaders who not only have a vision and can motivate others behind that vision, but who are 
also able to formulate strategy and translate that strategy into concrete priorities and specific action 
(Davies & Davies, 2005; Hughes & Beatty, 2005). 

Some scholars draw sharp distinctions among these approaches. For example, some researchers 
describe instructional leadership as "top-down" and transactional and instructional leadership as 
"bottom-up" and "relational" (Hallinger, 2003). Others have argued that these models are not mutually 
exclusive and that effective leaders employ multiple leadership approaches, pulling from a combination 
of skills and strategies, as the situation demands (Bass & Riggio, 2006; Marks & Printy, 2003). For the 
purposes of this report, we employ these models not as distinct leadership types but as dimensions 
through which to describe aspects of principal leadership as revealed in our first year of data. 

This chapter describes the principals leading the 25 core sample schools in spring 2011. Because of the 
emphasis on principal replacement in SIG, we begin with an analysis of how long these principals have 
been at their respective SIG schools and their prior leadership experience. We then consider the 
evidence from our first year of data on the qualities of leadership these principals demonstrated. We 
discuss the degree to which they demonstrated aspects of transformational and instructional leadership, 
drawing on teacher and instructional coach interviews, teacher and school improvement team focus 
groups, and teacher survey data. We then turn to data from the principals themselves to explore the 
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explicitly strategic (or nonstrategic) nature of their leadership as defined by the cohesiveness and clarity 
of their individual theories of action for their schools. 18 We end with a synthesis of the findings across 
the three leadership approaches considered: transformational, instructional, and strategic. Additional 
analysis of the associations between the leadership qualities of the 25 core sample school principals and 
other leading indicators, such as the school's overall capacity, is discussed in Chapter 7. 


Box 4.1. Key Chapter 4 Findings 

• In accordance with SIG guidelines, the majority of schools (21 of 25) replaced their principals in 
2009-10 or 2010-11 (one school did so twice). Most principals in the core sample (21 of 25) 
were experienced leaders, as they had experience being principals at their current post or at 
other schools; their estimated average experience being principals was 5.5 years. 

• When classified based on our three dimensions of leadership (transformational, instructional, 
and strategic), 2 of 25 principals placed high and 2 of 25 principals placed low on all three 
dimensions. The remaining 21 of 25 principals reportedly exhibited a mixture of these qualities. 
For example, some principals received high scores on one or two dimensions of leadership but 
medium scores on the other dimensions. 


Principal Replacement and Experience 

There are few well-designed randomized controlled trials or quasi-experiments supporting the claim 
that replacing the principal or altering leadership practices leads to school-level change (Herman et al., 
2008). Case study research, however, does suggest "that the principal is a key part of school change and 
turnaround," and that a change in the school leader may serve symbolic as well as substantive purposes 
in the turnaround process: "Because the current school leader may be enmeshed in past strategies, 
installing a new principal can signal change" (Herman et al., 2008, p. 11). Reflecting this hypothesis, SIG 
requires schools adopting either the turnaround or transformation model to replace the principal. The 
only exception allowed is for principals who have been at their respective schools for less than two 
years. Thus, this chapter begins by asking: Did the schools in this sample replace the principal, and what 
was the experience level of the new leader installed in these schools? To address this question, data 
were analyzed from both the principal and district interviews. 

Principal Replacement 

Principals in 11 of 25 core sample schools were hired in the first year of SIG (2010-11), and principals 
in 11 of 25 schools were hired just one year prior to SIG (2009-10). 19 Of the 22 principals hired during 
this two-year time period, 16 were hired from within the district. 20 Seven of the 22 principals mentioned 
that they voluntarily applied for the job, while 13 out of the 22 principals mentioned that they had been 
assigned by the district to their position or recommended for the job. For example, one principal shared 
the following: "I was assigned out here by the school board because the previous principal got a 


18 See Exhibits B.5-B.7 for a detailed discussion of the different data sources used for the principal leadership 
analyses. 

19 One school hired a new principal in 2009-10 and replaced this principal in 2010-11. 

20 This includes two restarts where principals were hired from within the network of the CMO that took over the 
school. It does not include information for one principal that was hired in 2009-10, because there was not 
sufficient information to make a classification. 
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promotion." Data regarding the process of principal replacement were unclear or ambiguous for 2 of the 
22 principals. 

The remaining four principals had been at their schools prior to 2009-10 for two or more school years. 

At three of these schools, the district decided to maintain the principals in their positions due to ongoing 
reform efforts or to avoid disrupting leadership at the schools. For example, the district administrator 
for one high school mentioned the following: "They've had the benefit of strong leadership capacity in 
instructional focus, and a lot of other factors.. .Staffing was strong even though we have work to do 
academically with instructional strategies. There wasn't the need to disrupt or change out the staff 
there. So, the transformation [model] seemed much more appropriate." 

There were no discernible patterns with regard to principal replacement and type of SIG improvement 
model being implemented. In accordance with SIG requirements, nearly all schools implementing the 
transformation and turnaround models replaced their principal either the year prior to SIG (in 2009-10) 
or during the first year of SIG (2010-11)— with the exception of the four schools noted previously. In 
addition, three of the restart schools, which were not required to replace their principal nonetheless did 
so in either 2009-10 or 2010-11. 

Principal Experience 

The relationship between principal experience and school performance is inconclusive. As part of a 
study exploring the role of principals in school outcomes, Clark, Martorell, and Rockoff (2009) conducted 
a literature review and found that few studies examine the relationship between principal effectiveness 
and years of experience, and that existing research points to mixed results. Nevertheless, researchers 
and policymakers hypothesize that principal experience may be an indicator of principal effectiveness 
(Rice, 2010). In the same nonexperimental study, Clark et al. (2009) examined the correlation between 
years of principal experience and school performance (as measured by student achievement) and found 
a positive relationship. The data from the study also indicated that schools with first-time principals who 
were previously assistant principals at their schools outperformed schools with first-time principals who 
did not have previous leadership experience at their school. 

Most core sample principals in Year 1 of SIG were experienced leaders. Across all 25 core 
sample schools, 21 principals had prior experience as principals either at their current position or at 
other schools. For these principals, their estimated average experience as principals was 5.5 years. 21 The 
other four principals (two in restart schools) were first-time principals in 2010-11. All of these first-time 
principals reported having been in a leadership or administrative position, such as assistant principal, 
prior to their appointment as principal, and all had been in education for more than five years. 

Across the 25 schools, 20 principals indicated they had prior experience working in low-performing 
schools and cited the role that they had played in turning around such schools. 

For example, one principal who has been in school administration for 20 years mentioned his experience 
turning around a school with a similar population: "We were functioning under the standards of any 
other school. When I arrived there it was bad, when I left we improved it to a high [grade]." Another 
principal reported that his experience increasing the student achievement of his previous school made 
him an attractive leader for SIG. He noted that he worked with the staff of his previous school to raise its 
rating by two levels. 


21 One of the 21 principals was excluded from this average because he did not report his number of years of 
experience, even though he stated that he had "many years" of experience as a principal and school administrator. 
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Respondent Perceptions of Principal Leadership Approaches 

Scholars have posited multiple dimensions of leadership, including transformational, transactional, 
instructional, distributed, and strategic leadership (Hallinger, 2003; Lynch, 2012). We focus on three of 
these leadership approaches relevant both to the data collected in our first-year site visits and to the 
context of persistently low-performing schools engaged in turnaround efforts: transformational, 
instructional, and strategic. We begin with teachers' assessments of their principals' leadership qualities 
with respect to transformational leadership and instructional leadership. We then turn to principal 
interview data and assess strategic leadership based on the principals' articulated theories of action. 

Transformational Leadership 

Transformational leadership theory emerges from the research on business organizations, public 
institutions, and the military, as well as research on schools. Coined by Burns in 1978, transformational 
leadership is a process by which "leaders and followers raise one another to higher levels of morality 
and motivation" (Burns, 1978, p. 20). Transformational leaders unite their staff around a vision or 
shared conception of the goals (Leithwood, 1994) and develop proactive and innovative leadership 
among others in the organization (Bass & Riggio, 2006). The transformational leadership construct 
assumes a distributive type of leadership in which the school leader understands "the needs of 
individual staff rather than 'coordinating and controlling' them towards the organization's desired ends" 
(Hallinger, 2003, p. 337). Researchers also hypothesize that transformational leaders "achieve a high 
level of trust" and that the "development of a culture of trust is deemed more effective than the short- 
term benefits of meeting predetermined targets through coercion or rewards" (Lynch, 2012, p. 3). In 
essence, this leadership model is characterized by leaders who inspire their staff, promote a shared 
vision for change, and focus on increasing their staff's capacity, commitment, and initiative (Hallinger, 
2003; Lynch, 2012). 

The analysis of transformational leadership qualities among the 25 core sample principals derives from 
two data sources: qualitative data (interviews with teachers and instructional coaches, focus groups 
with teachers and the school improvement team) and teacher survey data (principal-teacher trust 
scale). 

Qualitative data. We first examined coded responses to open-ended questions in which teachers, 
instructional coaches, and the school improvement team had an opportunity to describe leadership at 
their school. For example, the questions for the teacher interviews included the following: How would 
you characterize the leadership of this school? Who are the key leaders, and what do they do to move 
the school forward and support you as a teacher? 

We identified 19 principal qualities or practices that emerged across respondent descriptions. These 
qualities were then compared with the 21 qualities of effective leadership delineated by Waters et al. 
(2003) in their meta-analysis of principal leadership studies, and based on this comparison, we pared 
down the list of principal qualities to those consistent with Waters et al. To focus the subsequent 
analyses, we limited the group of qualities considered to those mentioned by at least two 
respondents— either a teacher, an instructional coach, or a member of the school improvement team— 
per school and in at least two schools per quality. The process resulted in a set of 11 qualities, 8 of 
which, in accordance with the research literature, are characteristic of transformational leaders while 3 
are uncharacteristic (see Exhibit 4.1 for more detail about the 11 qualities). If a quality of 
transformational leadership was cited by at least two school-level respondents (teacher, instructional 
coach, or member of the school improvement team), then the school's principal was coded as exhibiting 
that quality. 
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Survey data. The second source of data on transformational leadership is teacher survey responses to 
items measuring principal-teacher trust, another foundational element of transformational leadership. 
These items and the resulting scale were drawn from teacher surveys administered by the Consortium 
on Chicago School Research (CCSR), and data collected by the CCSR indicate that principal-teacher trust, 
as measured using the scale, is associated with improved student achievement (Bryk & Schneider, 2002). 
The items asked teachers to indicate the extent to which their principals displayed behaviors identified 
in the literature, such as taking a personal interest in the professional development of teachers or 
placing the needs of children ahead of personal and political interests, using a scale of 1 to 4 (see Exhibit 
B.5 for more detail on the analytic methods). 


Box 4.2. Perceptions of Transformational Leadership 

The classification of core sample school principals by perceptions of transformational leadership is 
described below (see Exhibit B.5 for more detail on the analytic procedures). 

High on Continuum 

• Qualitative data: The principal demonstrated at least three of the eight qualities related to 
transformational leadership (Demonstrated qualities refer to those mentioned by at least two 
respondents, which included teachers, instructional coaches, and members of the school 
improvement team) (see Exhibit 4.1); AND 

• Survey data: Principal-teacher trust scale average was at least 0.5 standard deviations (0.35) 
above the scale mean (3.09). 

Middle on Continuum 

• The principal did not demonstrate evidence to be categorized in the high or low end of the 
continuum. 

Low on Continuum 

• Qualitative data: The principal demonstrated none of the eight qualities related to 
transformational leadership (see Exhibit 4.1); AND 

• Survey data: Principal-teacher trust scale average was at least 0.5 standard deviations (0.35) 
below the scale mean (3.09). 

Exhibit 4.1 reports the number of principals displaying each of the eight transformational leadership 
qualities: being accessible/welcoming of input, being supportive of staff, being visible/known to school 
community, being a visionary, being enthusiastic, being communicative, able to develop leaders, and 
having high expectations. Exhibit 4.1 also reports the number of principals displaying the three qualities 
that are contrary to transformational leadership: having poor communication, being unsupportive of 
staff/having bad rapport, and being authoritarian. 
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Exhibit 4.1. 

Number of Principals with Reported Leader Qualities Characteristic and 
Uncharacteristic of Transformational Leadership in Core Sample Schools 


Reported Leader Qualities Characteristic of Transformational Leadership 

Number of Principals 

Accessible/Welcomes Input 

13 

Supportive of Staff 

11 

Visible/Known to School Community 

7 

Visionary 

6 

Enthusiastic 

4 

Communicative 

4 

Develops Leaders 

2 

High Expectations 

Reported Leader Qualities Uncharacteristic of Transformational Leadership 

2 

Poor communication 

5 

Unsupportive of Staff/Bad Rapport 

3 

Authoritarian 

3 


Source: SST teacher and instructional coach interviews, and teacher and school improvement team focus groups, spring 2011. 
Note: Includes 23 of 25 core sample schools. Two schools were excluded from this analysis due to insufficient qualitative data. 


About one fifth (4 of 19) of core sample principals were at the high end of the continuum for 
exhibiting multiple qualities associated with transformational leadership. 22 According to teachers, 
instructional coaches, and members of the school improvement team, these principals demonstrated 
three or more of the eight qualities related to transformational leadership and had an average score of 
half a teacher-level standard deviation (0.35) or more above the overall mean (3.09) on the principal- 
teacher trust scale. Based on the information provided by teachers, instructional coaches, and members 
of the school improvement team, two of these four core sample principals reportedly demonstrated 
transformational leadership qualities primarily related to the relational aspects of being a principal- 
such as, being accessible and welcoming of input, being supportive of staff, being enthusiastic, and being 
communicative. For example, a teacher from one of these schools said: "You can walk into [the 
principal's] office and say 'I have this idea.' The open door policy is definitely here. ..e-mail, face-to-face, 
however you feel comfortable doing it is how you can do it." An instructional coach at the school also 
said: "I think [the principal] as a leader is incredible. He sets a terrifically optimistic tone." 

Meanwhile, teachers, instructional coaches, and members of the school improvement team suggested 
that the other two principals demonstrated a range of the transformational leadership qualities, 
including both relational qualities and other transformational leadership qualities, such as being 
visionary, developing leaders, and having high expectations. A teacher from one of these schools 


22 Six core sample schools were excluded from this analysis for the following reasons: (1) four schools did not meet 
the survey response threshold of 50 percent or above, but qualitative data show that two of these four principals 
demonstrated at least three of the eight qualities related to transformational leadership; and (2) two core schools 
had insufficient qualitative data— one of these schools had faulty audio files so we were unable to transcribe and 
code the interviews— and a review of the coded data for the second school revealed that none of the teachers, 
instructional coaches, or members of the school improvement team discussed the principal's leadership despite 
having an opportunity to provide that information. 
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described the relational aspects of the principal's leadership style: "[The principal] will discuss [changes] 
and ask us to chime in. [There is] lots of e-mail communication. In leadership meetings, we'll discuss 
what needs to stay in-house and what needs to be disseminated." However, these respondents also 
described other aspects of the principal's leadership style. One teacher said: "[The principal] has a vision 
for where we're going, and she's willing to let staff know that. She doesn't give all at one time. She's 
good at breaking it down. ..and not overwhelming staff." 

About another fifth (4 of 19) of core sample principals were categorized as exhibiting none of the 
qualities associated with transformational leadership. These four principals received principal-teacher 
trust scores that were half a teacher-level standard deviation (0.35) or more below the overall mean 
(3.09), and fewer than two respondents at these four schools described their principal as exhibiting any 
of the eight transformational leadership qualities. In addition, in three of these schools, two or more 
respondents described their principal as exhibiting at least one of the qualities that are uncharacteristic 
of transformational leaders (see Exhibit 4.1). A teacher at one of these schools noted: "It got really 
difficult this year. The decision-making process was kind of jolted and halted because [the principal] was 
throwing down these mandates. ..and it turned into arguing and fighting. It was two or three years ago 
that we were still making decisions as a staff." An instructional coach at the other school described the 
principal in the following manner: "It was really challenging. From the beginning, [the principal] had a 
deficit view of teachers and all the staff rather than an assets view... .For coaching, he had a thing where 
we would report back to him. ..and tell him who is good and who is bad rather than trying to build 
people up." 

The remaining majority (11 of 19) of core sample principals reportedly exhibited a mix of 
transformational leadership qualities. Seven of these principals were described by at least two 
respondents as exhibiting at least one quality related to transformational leadership and had average 
scores close to the overall mean (3.09) but not more than half a standard deviation (0.35) above or 
below the overall mean on the principal-teacher trust scale. According to teacher reports, two other 
principals demonstrated at least two qualities related to transformational leadership. These two 
principals had averages on the principal-teacher trust scale that were approximately half a teacher-level 
standard deviation below the overall mean (2.7 and 2.8). The remaining two principals did not receive 
any reports of any transformational leadership qualities from their staff but scored within half a 
standard deviation (0.35) above or below the overall mean (3.09) on the principal-teacher trust scale 
from the teacher survey (3.0 and 3.3). 

Instructional Leadership 

Instructional leadership focuses on efforts to improve the technical core of schooling: teaching and 
learning. Instructional leadership is often described as "strong, directive leadership focused on 
curriculum and instruction from the principal" (Hallinger, 2003, p. 329). The focus in much of the 
literature is on the principal as the leader guiding and monitoring instruction, creating and managing the 
curriculum and instructional program (Bossert, 1982; Hallinger, 2003; Leithwood, 1997; Lynch, 2012). 
Other qualities that characterize instructional leadership, which may be either transformational or 
transactional in character, 23 include defining the school mission and promoting a positive learning 


Burns (1978) distinguishes two general approaches to leadership: transactional and transformational. 
Transactional leadership focuses primarily on "extrinsic motivations" and focuses on exchanges— a principal gives 
teachers support in exchange for the desired behavior (Burns, 1978; Lynch, 2012). Transformational leadership is 
not characterized by these types of exchanges but is instead motivated by beliefs and values. While some scholars 
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climate— which may involve promoting professional development and maintaining high visibility 
(Hallinger, 2000). 

As in the case of transformational leadership, we used both qualitative and survey data to explore 
principals' instructional leadership. (See Exhibit B.6 for more detail on the analytic methods.) 


Box 4.3. Perceptions of Instructional Leadership 

The classification of core sample principals on perceptions of instructional leadership is described below 
(see Exhibit B.6 for more detail on the analytic procedures). For this analysis, "respondents" come from 
the following respondent groups: teachers, instructional coaches, and school improvement teams. 

High on Continuum 

• Qualitative data: At least two respondents described principal behaviors associated with 
instructional leadership (e.g., being focused on academics and academic achievement of 
students, providing feedback on instruction and/or curriculum, observing and being visible in 
the classroom, and being involved in professional development, instruction, and coaching); AND 

• Survey data: Instructional leadership scale average was at least 0.5 standard deviations (0.33) 
above the scale mean (3.12). 

Middle on Continuum 

• The principal did not demonstrate evidence to be categorized in the high or low end of the 
continuum. 

Low on Continuum 

• Qualitative data: Fewer than two respondents mentioned principal behaviors associated with 
instructional leadership; AND 

• Survey data: Instructional leadership scale average score was at least 0.5 standard deviations 
(0.33) below the scale mean (3.12). 

About one-sixth (3 of 19) of core sample principals demonstrated instructional leadership. According 
to teacher, instructional coach, and school improvement team reports, these principals exhibited the 
features of leadership defined by the instructional leadership model. All three principals received 
average survey scores on the instructional leadership scale that were at least half a teacher-level 
standard deviation (0.33) above the overall mean (3.12). For one of these principals, a teacher said: 
"Whatever we're working on, [the principal] is involved. She attends all of our team meetings, she is in 
our classrooms, and she calls herself our instructional leader. I like how specific she is when she comes 
into our classrooms." Another teacher at the same school said: "It's been helpful to. ..change the vision 
of the principal from a manager to an instructional leader." At a different school, a teacher stated: "The 
principal... really picked apart our curriculum and asked us: what is working? What is making teaching 
here work for you? What is making your kids be successful? And, what is not working so that we can get 
something on board to fix that?" 

About another one-sixth (3 of 19) of the core sample principals were not described as instructional 
leaders by respondents. These principals also had average scores that fell half a teacher-level standard 
deviation (0.33) or more below the overall mean (3.12) on the instructional leadership scale. 


describe instructional leadership as primarily transactional (Hallinger, 2003), others emphasize a more distributed 
version of instructional leadership (Elmore, 2000; Spillane, Flalverson, & Diamond, 2004). 
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The remaining majority (12 of 19) of core sample principals were categorized in the middle of the 
continuum on instructional leadership. These principals were not described as instructional leaders by 
school-level respondents 24 but had average scores that fell within half a teacher-level standard deviation 
(0.33) above or below the overall mean (3.12) on the principal instructional leadership scale. 

Strategic Leadership 

Thus far we have considered principal replacement and experience in the first year of SIG, as well as 
respondent reports from teachers, instructional coaches, and school improvement team members on 
the qualities of leadership exhibited by those principals in 2010-11. In this section, we look at how the 
principals themselves conceptualize the change process at their schools and the priorities they put in 
place to guide reforms. Our data provide a description of the extent to which the core sample principals 
engaged in strategic leadership of the turnaround process, as well as a baseline description of the 
principal's strategy, to be used in future analyses of how strategies evolve over time as principals 
interpret and learn from their actions and results. Our analysis is based on data on the principals' 
perceptions about the current condition of the school and also about where the school is headed and 
the concrete actions that will move it in the desired direction. 

One way to analyze this aspect of principal leadership is to focus on principals' organizational "theories of 
action" (Argyris & Schon, 1974). Theories of action are essentially "stories of how a problem arises and 
how problems can be solved" (Weiss, 1995). They are the underlying assumptions held by stakeholders 
about how they will bring about change in their schools— articulations of how they will get from their 
existing condition to their intended destination. Even though research on the relationship between 
theories of action and improved student outcomes is limited and inconclusive, researchers hypothesize 
that there may be value in having a theory of action to guide school change (City et al., 2009; Fullan, 2007). 
A common assumption is that encouraging stakeholders to explicitly articulate a theory of action will bring 
to light underlying assumptions about the improvement efforts, which will then help to guide the change 
process further (Argyris & Schon, 1974). SIG principals work in complex systems— in addition to SIG 
requirements, the schools in which they operate may have other ongoing district, state, or federal 
initiatives that they may have to follow. One purpose of a good theory of action is to help the school 
personnel find "a clear path through this initiative thicket" (City et al., 2009, p. 45) and to identify the key 
improvement actions that need to be implemented to improve teaching and learning. 

Although the theory of action concept may be useful for understanding the mental maps guiding 
improvement efforts at a school, it has certain limitations. Principals' theories of action are not 
necessarily representative of the entire school or district— there may be multiple and even contradictory 
theories of action within a system (Weiss, 1995). Theories of action are supposed to facilitate reflection 
and change (City et al., 2009), which means that they may shift from time to time. The theories of action 
discussed below represent those held by principals at the point in time in which they were interviewed 
and may have since been revised or deepened. 

We relied on transcribed principal interviews as the primary data source for this analysis, which 
proceeded in four stages: 


24 One principal was excluded from this count. This principal had a score that was almost an entire standard 
deviation above the overall mean on the instructional leadership scale. However, there were no qualitative data 
from the school respondents— teachers, instructional coaches, and members of the school improvement team— to 
support this finding despite the fact that respondents had the opportunity to provide information on this topic 
during interviews and focus groups. 
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Stage one. Analysts developed an operational definition of a principal's "theory of action" to ensure 
clear and consistent use of the term and to uncover the components of a theory of action. A theory of 
action, broadly stated, is the implicit or explicit set of operational assumptions regarding how the 
change process will unfold in a given school. Following from this definition, analysts identified five 
elements of a theory of action: (1) defining the performance problem, (2) identifying a set of 
improvement actions or primary levers of change to address the performance problem, (3) providing a 
rationale for selecting those improvement actions, (4) identifying the intended outcomes of those 
strategies, and (5) explaining the explicit and interrelated assumptions underlying the change process in 
a school. 

Stage two. Using this five-element framework, the analysts coded the principal interviews to capture 
which of these five elements of a theory of action were present in the principal's interview. Although 
principals were not explicitly asked to describe their theory of action, principals did have the 
opportunity to discuss each of the five elements of a theory of action. For example, principals were 
asked: Why do you believe this school has remained low performing year after year? What has hindered 
improvement efforts in the past? What do you think your school needs to improve student 
performance? What is your main priority as principal? These types of questions allowed principals to 
discuss how they define the performance problem and to discuss the improvement actions or levers of 
change that they planned to implement to address the perceived performance problem. Throughout the 
interviews, principals also had the opportunity to provide rationales for selecting specific improvement 
actions, explain the intended outcomes of the selected improvement actions, and explain their 
assumptions about the change process taking place at the school. 

Stage three. Having coded the principal interviews using the five-element framework described 
previously, the study team summarized each principal's approach to capture an overall narrative for 
each principal's theory of action. The process of synthesizing the elements of a principal's theory of 
action allowed analysts to condense the coded data and facilitated the analysis on how a principal 
conceptualized the change process at his or her school. 

Stage four. To examine the extent to which principal responses revealed a theory of action, the analysts 
then categorized the theory of action summaries to determine the degree to which principals held a 
theory of action (see Exhibit B.7 for a more detailed discussion of the analytic methods). 
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Box 4.4. Perceptions of Strategic Leadership: Theories of Action as Reported 
by Principals 

The classification of core sample principals based on theories of action is described below (see 

Exhibit B.7 for more detail on the analytic procedures). 

High on Continuum 

• The principal exhibited at least four of the five elements of a theory of action (i.e., defining 
the performance problem, identifying a set of improvement actions or primary levers of 
change to address the performance problem, providing a rationale for selecting those 
improvement actions, identifying the intended outcomes of those strategies, and explaining 
the explicit and interrelated assumptions underlying the change process in a school); AND 

• The principal expressed that these elements mutually reinforce each other. 

Mid-High of Continuum 

• The principal exhibited at least three of the five elements of a theory of action; AND 

• The principal expressed that some of these elements mutually reinforced one another, but 
not all of the elements reinforced one another or related to the perceived performance 
problem. 

Mid-Low on Continuum 

• The principal exhibited fewer than three of the five elements of a theory of action; AND 

• The principal expressed that most of these elements were externally driven and most of the 
improvement strategies did not mutually reinforce one another or address the perceived 
performance problem. 

Low on Continuum 

• The principal exhibited fewer than three of the five elements of a theory of action; AND 

• The principal expressed some plans for school improvement but was not able to or did not 
articulate many components of the plan. It was not evident that strategies mutually 
reinforced each other. 


One fifth (5 of 25) of the core sample principals articulated a theory of action, and were thus at the 
high end of the continuum. These principals were able to describe at least four of the five elements of a 
theory of action. For example, the principal at one school first determined that the performance 
problem at the school stemmed from poor teacher quality and lack of follow-through of instructional 
programs. The principal stated: "There need to be programs done with fidelity; there need to be quality 
teachers. Once you have these things in place you can talk about professional development activities 
and how to attain proficiency with low-performing students." The principal articulated several primary 
levers of change to address these concerns, such as: replacing staff members based on whether they 
had demonstrated willingness to improve 25 and ensuring follow-through of instructional programs by 
increasing visibility of schoolwide programs during common planning time with the teachers and 
establishing daily walkthroughs to check for consistency in program implementation. Teachers were also 
given the opportunity to collaborate with their peers and receive support and guidance from 


25 Some teachers also voluntarily left. 
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instructional coaches for the implemented programs. Additionally, the intended outcomes and the 
rationale behind these strategies as articulated by the principal directly served the two goals of 
improving teacher quality and ensuring follow-through or consistency. 

Another 8 of 25 core sample principals identified several aspects of their theory of action, but some 
aspects were not evident, and so they fell in the mid-high range of the continuum. These principals 
demonstrated in their interviews at least three of the five elements of a theory of action, but certain 
components of their plan were not fully articulated or were unrelated to the perceived performance 
problems at the school. For example, the principal at one school noted that the school's performance 
problem stemmed from poor school leadership in previous years, a culture of low expectations, and 
contextual factors, such as high rates of poverty and crime in the community. The principal also shared 
elements of a plan regarding the change process: the priority was to first improve safety and then focus 
on instruction. She identified specific strategies— such as replacing ineffective teachers— that she 
believed addressed some of the perceived performance issues, such as the culture of low expectations. 
However, the principal did not provide a rationale of how the remaining improvement actions related to 
the overall goals of the school. For example, she mentioned the use of instructional coaches and 
extending the school day but did not directly link these to the identified performance problem or school 
improvement goals. 

Three of these eight principals communicated a plan with regards to the implemented improvement 
actions, but they did not share their perceptions of why the school had a history of low performance. 
One of these principals deflected the question, and another principal talked about issues that he could 
not address, such as the contextual factors of the school (e.g., high rates of crime, unemployment, and 
mobility in the neighborhood). In the third case, the principal had just arrived in the school and indicated 
that she could not speak to the issues that had contributed to the school's history of low performance. 
However, research points to the importance of understanding the performance problem at a school 
since this will help determine the necessary change strategies (Duke & Salmonowicz, 2010). Because 
these three principals did not articulate an understanding of the performance problem, it was difficult to 
determine whether or not the improvement plans they set forth addressed the perceived issues at the 
school and thus whether these actions were mutually reinforcing and strategic. 

Unlike the five principals on the high end of the continuum, whose responses revealed a theory of 
action, the principals in this category did not consistently provide a rationale for their decisions to 
implement certain strategies or how those strategies related to overall school improvement efforts. 

The remaining 12 of 25 core sample principals did not articulate a theory of action, and so were placed 
in the mid-low (9 principals) and low (3 principals) ends of the continuum. The nine mid-low principals 
mentioned improvement actions going on at their school but viewed these actions as primarily 
externally driven and did not relate the actions to an overall plan for school improvement. In their 
interviews, these principals generally discussed less than three of the elements of a theory of action. 

Five of these nine principals did not report being a key player in the change process. For example, one of 
these principals mentioned that the main goals of the school, such as focusing on reading, mathematics, 
and graduation rates, had been set by the district. Even though some of the improvement actions 
seemed to align with the goals set forth by the district, the principal did not articulate a strategy. 

Another principal mentioned that the district had implemented a science and mathematics theme at the 
school as well as various strategies related to that initiative, but the principal never indicated how this 
related to improvement efforts going on at the school. And in a third case, the principal explicitly stated 
that the implemented actions were primarily compliance-driven. This principal was essentially doing 
what she was told to do, without actually having a conception of the problems at the school or a 
strategic approach to addressing those problems. The actions being implemented in these schools, 
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however, were not necessarily negative— some of those mentioned by the principal, such as 
professional development or increasing use of time, may be potentially useful mechanisms for 
supporting school turnaround (Duke, 2006). But the data from these principal interviews revealed that 
these principals had not made sense of these strategies or related them to the needs of their school. 
They had not developed a theory of action that cut through the "initiative thicket" (City et al., 2009). 

Three principals in this group of nine also talked about the strategies in their schools as externally 
driven, but their comments revealed an intention to take greater ownership over the improvement 
actions and improvement plan at the school. Those intentions had borne little fruit as of spring 2011, 
however. The alternative plans these principals set forth were not connected to the performance 
problem, and it was evident that they also believed the improvement actions were externally driven. For 
example, in one of these schools, the principal reported plans for improvement but also mentioned that 
she did not really have the jurisdiction to implement the changes she felt were needed to improve the 
school. The principal said: "The model is 'transformation,' but we haven't transformed anything. We've 
gotten more money for technology, but we need to change things." Finally, one other principal reported 
improvement actions that were both externally driven and not related to the perceived performance 
problem at the school. 26 

The three principals in the low range of the continuum articulated elements of a plan but reported not 
being able to fully implement it because of special circumstances that distracted them from their main 
goals. For example, one principal reported that the school deals with a lot of surrounding community 
violence— the main issues at the school, according to the principal, were student discipline, parent 
attitudes, a culture of low expectations, and poor instruction. His priorities were establishing order in the 
school, creating a safe environment, and instilling a culture of high expectations. Flowever, the principal 
noted that in 2010-11, she spent most of the time "putting out fires." The principal at this school 
recognized the need to eventually address instruction, saying: "We need to be able to look at instruction 
instead of being in survival mode of getting to the next day." Another principal articulated an 
improvement plan and strategies for how to achieve that plan, but at times the actions taken by the 
principal seemed to contradict the overall goals for improvement. For example, this principal mentioned 
that ineffective teachers and a toxic school climate were contributing to the low performance of the 
school. The principal suggested that professional development and teacher replacement would address 
this issue, yet he reported that he had switched teacher assignments to make staff uncomfortable and 
more likely to voluntarily leave the school. This strategy backfired and created a negative political climate 
within the school that distracted the principal from his plan for improvement. The third principal in this 
group had only recently arrived at the school and was not planning to stay in that role. Flowever, this 
principal did articulate a vision and mentioned some improvement actions that he would have liked to 
implement for the remainder of the term. 

Together these data indicate that approximately half (13 of 25) of the core sample principals had 
developed improvement strategies that both identified and systematically addressed perceived 
performance problems of the school, but in only five of these schools were the strategies well- 
articulated and connected. The remaining 12 principals may have had a few elements of a strategy, but 
analysts could not identify an underlying theory of action guiding the improvement actions. 


Note that this principal's interview transcript was not fully transcribed due to poor audio quality. 
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Chapter Summary 

This chapter examined principal leadership using three lenses derived from the empirical literature: 
transformational, instructional, and strategic leadership. To explore the extent to which principals 
engage in more than one approach to leadership, we tallied the number of core sample principals who 
engaged in each of the three approaches to leadership in Exhibit 4.2, which also indicate the principal's 
reported years of experience as a principal. Two principals rated high on all three dimensions 
(transformational, instructional, and strategic), and two principals rated low on all three dimensions. 
Several other principals rated high on one or two approaches only. For example, the principal at Sterling 
Slope Elementary rated high on transformational and strategic leadership, while the principal at Haven 
Way Elementary rated high on instructional and strategic leadership. 

In general, it appears that how a principal rates on one aspect of leadership is predictive of how he or 
she will rate on the other two aspects of leadership. This apparent pattern suggests that the three 
dimensions of leadership overlap, and that although the models may be distinct theoretical constructs, 
in practice principals may pull strategies and approaches from these and other leadership models. 

There does not however appear to be a strong association between a principal's level of leadership and 
the principal's level of experience as a principal. For example, two of the four principals rated most 
highly on leadership overall were first-time principals while the other two already had 5-10 years of 
experience as a principal. Likewise, among the three principals with the lowest leadership ratings, one 
had only 1-4 years of experience as a principal, another had 5-10 years of experience, and the third had 
more than 10 years of experience. In the next two years of data collection for this study, we will 
continue to delve into the role of the principal in these 25 schools, further exploring aspects of 
leadership introduced in this chapter. 
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Exhibit 4.2. 

Summary of Leadership Dimensions, by Core Sample School 


School 

Transformational 

Leadership 

Instructional 

Leadership 

Strategic 

Leadership 

Years of Experience as 
a School Principal 

Baltimore Bridge Elementary 

High 

High 

High 

0 

Rossignol Elementary 

High 

High 

High 

5-10 

Haven Way Elementary 

High 

Medium 

High 

0 

Sterling Slope Elementary 

High 

Medium 

High 

5-10 

Sherbrooke Elementary 

Medium 

Medium 

High 

5-10 

Proctor Point High 

High 

Medium 

Mid-High 

1-4 

Big Acorn High 

Medium 

Medium 

Mid-Low 

5-10 

Coral High 

Medium 

Medium 

Mid-Low 

5-10 

Elmsville High 

Medium 

Medium 

Mid-Low 

5-10 

Gillepsie High 

* 

* 

Mid-High 

5-10 

Inner Brooks High 

Medium 

Medium 

Mid-High 

0 

McAlliston High 

* 

* 

Mid-High 

1-4 

Melon Elementary 

Medium 

Medium 

Mid-Low 

5-10 

Peregrine Hill Elementary 

* 

* 

Mid-Low 

5-10 

Paul Bunyan High 

Medium 

Medium 

Mid-Low 

1-4 

Sawbuck Elementary 

Medium 

Medium 

Mid-High 

0 

Meribel High 

* 

* 

Mid-High 

1-4 

West Marble High 

Medium 

Medium 

Mid-High 

1-4 

Aerovista High 

Medium 

Low 

Mid-Low 

** 

Blizzard Bay Elementary 

Low 

Medium 

Mid-High 

5-10 

Gale Secondary 

* 

* 

Mid-Low 

1-4 

Raven Ridge Elementary 

* 

* 

Low 

5-10 

Tyro Trail Elementary 

Low 

Medium 

Mid-Low 

5-10 

Island Bay Elementary 

Low 

Low 

Low 

>10 

Tyron Elementary 

Low 

Low 

Low 

1-4 


Source: SST teacher and instructional coach interviews and teacher and school improvement team focus groups, spring 2011; 
SST teacher survey, spring 2011. 

Notes: All school names are pseudonyms. Cells are shaded to help illustrate potential patterns between the principal leadership 
and experience measures. Green cells correspond to high levels of leadership and experience (i.e., at least 5 years of experience 
as a principal). Yellow cells correspond to medium levels of leadership and experience (i.e., 1-4 years of experience as a 
principal). Red cells correspond to low levels of leadership and experience (i.e., 0 years of experience as a principal). 

*These six schools either did not meet the 50% teacher survey response rate threshold or did not have sufficient qualitative 
data, so they were excluded from the analysis of transformational and instructional leadership. 

**Response was "many years of experience" but no actual number was reported. 
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Chapter 5: Improvement Actions in SIG Schools 


Although all 25 core sample schools were among the persistently lowest-achieving schools in their 
states, they differed in the neighborhoods in which they were situated, the resources available to them, 
the ways they defined the performance problem, the leadership of their principals, and the SIG models 
they implemented. These differences suggest that the schools could likewise differ in the actions they 
adopted during the first year of SIG. On the other hand, because SIG prescribes a set of requirements 
and expectations, the schools could share many commonalities, particularly for those implementing the 
same intervention model. For example, we might expect actions to vary across schools depending on 
the specific intervention model implemented and how prescriptive a particular SIG requirement was. 
Turnaround schools were the only ones required to replace at least 50 percent of their teachers in the 
first year of the grant. Although increased learning time was required in turnaround and transformation 
schools (but not for restart schools), SIG regulations were less specific about how to meet this 
requirement. Other activities such as "implementing approaches to improve school climate and 
discipline" were permissible in turnaround and transformation schools, but not required (U.S. 
Department of Education, 2010b). 

This chapter examines the improvement actions in the core sample schools during the first year of SIG, 
as reported by school and district respondents. The chapter begins with a brief overview of the activities 
these schools implemented to improve their performance, followed by an analysis of how schools 
budgeted their SIG funds to support these activities. The final sections of the chapter focus in more 
depth on three specific improvement activities: teacher replacement, extended learning time, and 
programs or polices intended to improve student behavior. We limited the number of activities 
examined to allow for more detailed discussion of their implementation. 

We chose these three activities for several reasons. First, they appeared in a substantial proportion of 
the core sample schools irrespective of the intervention model. Second, they each correspond to one of 
the three domains of activity outlined in SST's conceptual framework (human capital management, 
technical core, and conditions to support teaching and learning). Third, the level of prescriptiveness in 
the SIG requirements for each of these three activities varied and thus could result in differing patterns 
of implementation, which could be instructive for our subsequent analyses of the change process in SIG 
schools. With respect to teacher replacement, for example, the SIG requires turnaround schools to 
replace a specified minimum proportion of the teaching force (50 percent). With respect to extended 
learning time, SIG requires this action in turnaround and transformation schools, but does not specify 
either the amount of time or the format in which the extension is to occur. Finally, with respect to 
activities to improve student behavior, SIG permits but does not require such activities and is silent on 
their form or purpose. We expected the more prescriptive policies to result in more uniform compliance 
and the less prescriptive ones to result in greater variability across schools. 
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Box 5.1. Key Chapter 5 Findings 

• Eleven different improvement actions were commonly reported in the core sample schools in 
2010-11. The most commonly reported improvement action was increasing professional 
development (23 schools). To be included, individuals from at least three respondent groups in 
each school must have mentioned the improvement action. 

• Respondents at all of the turnaround and transformation schools that replaced at least half of 
their teachers during the first year of SIG (six turnaround and one transformation) reported 
that principals removed staff who were perceived to be less skilled or motivated. 

• SIG requirements, rather than an identified school need, appeared to be the main impetus for 
increasing learning time in the schools that did so. At 14 of the 20 schools that increased 
learning time in 2010-11, respondents indicated that the schools implemented these programs 
to comply with SIG requirements. 

• In contrast, although SIG schools were not specifically required to implement programs and 
policies to improve student behavior during school hours, 20 core sample schools reported 
implementing such programs and policies. 


School Improvement Actions in Core Sample Schools in 

2010-11 


This section provides an overview of the actions implemented and budgeted in the core sample schools 
during the first year of SIG, illustrates the scope of the schools' improvement actions, and describes how 
the schools planned to alter their performance over the course of the grant. 

Improvement Actions in Core Sample Schools 

During interviews and focus groups with respondents (principals, teachers, coaches, and district staff), 
site visitors asked about the improvement strategies and actions 27 taking place during the 2010-11 
school year. The site visitors did not query directly about specific strategies, such as increasing students' 
learning time, but rather gave respondents the opportunity to report on the actions being implemented 
as they defined and understood them. We focused only on what the schools were reportedly doing and 
did not collect comparably systematic information on the actual quality of the actions or depth of 
implementation. 

Analysts identified 11 different improvement actions in the core sample schools during the 2010-11 
school year. These 11 actions fall into the three domains outlined in SST's conceptual framework: 
human capital management, the technical core, and the conditions that support teaching and learning 
(see Exhibits 1.2 and 5.1). 


27 We asked respondents about the "strategies and actions" they were engaged in as part of their improvement 
efforts. When reporting on what the respondents said that they were doing to improve, the term "actions" rather 
than "strategies" is used, as "strategy" implies a plan of action to achieve a specific goal. Because SST is in part 
exploring the connection between the activities in these schools and the respondents' perceptions of the 
performance problem and their goals, we do not want to imply that the actions of the schools are necessarily 
strategic. 
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Box 5.2. Improvement Actions Implemented by Core Sample Schools 

Identification criteria for improvement actions are described below (see Exhibit B.8 for more detail on 
the analytic procedures). For this analysis, "respondents" come from the following respondent groups: 
district administrators, principals, teachers, instructional coaches, school improvement teams, external 
support providers, and parents. 

Identified as being implemented 

• Individuals from at least three respondent groups identified the improvement action as being 
implemented. For one improvement action (use of an instructional coach), two respondent 
groups were used as the threshold if analysts considered the respondents reporting the action 
to be key informants for the particular action (e.g., teachers and a coach reporting on coaching) 
or if analysts deemed it unreasonable for other school or district respondents to have deep 
knowledge about the action in question. 

Individuals from at least three respondent groups in 23 schools described an increase in professional 
development activities provided during the school day, in the summer months, or on specific 
professional development days. Replacing the principal and increasing learning time (both of which 
were required for most SIG schools) were also mentioned by at least three respondent groups in 22 core 
sample schools. On the other hand, the provision of parent engagement activities and the use of 
technology were each reported by at least three respondent groups in only 9 schools. 

There were no core sample schools where revised teacher or principal evaluation strategies were 
mentioned by at least three respondent groups, even though we might have expected a focus on these 
topics, at least in schools from states that received Race to the Top grants. The absence of a focus on 
teacher evaluation may be a matter of timing, and teacher evaluation could arise as a targeted action in 
the second or third year of SIG. Alternatively, it could be that respondents do not see teacher evaluation 
systems as part of the school improvement process but rather as a matter of district and state action 
only. 

The 11 actions summarized in Exhibit 5.1 correspond to the actions required or permissible in the SIG 
guidance. For example, the turnaround and transformation models require schools to "provide staff 
ongoing, high quality, job-embedded professional development" (U.S. Department of Education, 2010b). 
Replacing principals, replacing teachers, using student-level data, and increasing learning time are all 
mentioned in the SIG guidance as either required or permissible strategies. However, aside from teacher 
replacement, the implementation of these improvement actions did not appear to vary by SIG model 
(not shown). 

Two additional findings are summarized in Exhibit 5.1. First, the 11 activities span all three domains in 
SST's conceptual framework, although fewer schools appear to have implemented actions related to 
improving the conditions for teaching and learning as part of their improvement efforts. Second, some 
of the schools reporting a given action also indicated that this action had started prior to receiving SIG. 
This was the case for almost all reported actions (10 of 11). For example, 10 of the 23 core sample 
schools that reported increasing professional development also reported that they began instituting this 
action prior to 2010-11. The remaining 13 schools reported that such increases began occurring during 
the first year of SIG, although concurrence does not necessarily mean that SIG paid for or even catalyzed 
the action. In contrast, all nine of the core sample schools reporting use of technology as one of their 
improvement actions also reported that this action was initiated concurrent with SIG. In this and many 
of the other cases, respondents reported that districtwide initiatives or school-defined improvement 
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plans drove the efforts around improvement actions. We discuss the role of SIG and the timing of 
improvement actions further in Chapter 6. 


Exhibit 5.1. 

Number of Schools Adopting Specified School improvement Actions in 
Core Sample Schools, 2010-11 


Human Capital Management 

Increase professional development 

Replace principal 
Use instructional coach 
Replace teachers 

Technical Core 

Increase learning time 

Change core curriculum/instruction 
Use student-level data 
Provide student supports 
Use technology 

Conditions that Support Learning 

New behavior policies/programs 

Provide parent activities 



0 5 10 15 

Number of Schools 

■ Initiated Prior to SIG Initiated Concurrent with SIG 


20 


25 


Source: SST respondent interviews and focus groups, spring 2011. 
Notes: Includes 25 core sample schools. 


Across all three domains, each core sample school reported on average 6 distinct improvement actions 
in the first year of SIG, with one school reporting 3 and one reporting 11. For example, respondents in 
one school reported implementing the following 8 improvement actions: (1) hired a new principal in the 
2010-11 school year; (2) used four coaches (one each for reading, mathematics, science, and response 
to intervention) to improve teachers instruction; (3) extended the regular school day by one hour three 
days a week; (4) implemented professional learning communities, which met for a half-day each month 
(funded by SIG), to provide teachers with embedded professional development; (5) continued a positive 
behavior support program to address student behavior issues; (6) adopted a new mathematics 
curriculum; (7) provided wraparound supports to students; and (8) worked to engage parents in the 
school community by holding a series of events, including family literacy, mathematics, and science 
nights, and regular English classes led by a part-time parent liaison. 

Schools may vary in the number of reported actions for several reasons, and having a greater number of 
reported improvement actions should not be interpreted as being further along in the turnaround 
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process or having a greater likelihood of long-term success. One rationale for engaging in a smaller set 
of activities could be that the district or school principal identified a need to focus on only a few things 
in the first year to create "early wins" to motivate teachers or create necessary conditions for further 
reforms. This appeared to be the case at one core sample school where the principal and teachers 
indicated that students' challenging behavior had to be addressed before focusing on other important 
issues, such as using data to guide teachers' instruction. In addition, contextual factors outside the 
control of the school may have influenced how much could be put in place during the first year of SIG. 
For example, principals and district administrators for 12 schools reported that funding delays 
forestalled implementation of strategies identified in their SIG applications. Respondents in two of these 
schools reported that, because of the funding delays, they were unable to increase the number of hours 
in the school day in 2010-11 but planned to do so the following school year. 

Improvement Actions Included in SIG Budgets 

School budgets for the first year of SIG provide another source of information about schools' 
improvement actions. 28 We now explore the amount of SIG funding budgeted by each core sample 
school for each improvement action in 2010-11 and compare that to the improvement actions 
described by site visit respondents. Exhibit 5.2 summarizes the actions identified in core sample schools' 
Year 1 budgets based on the coding of each school's budget line items, as described in Chapter 2. 

Many of the improvement actions described by respondents are reflected in SIG budgets. For example, 
professional development was the most frequently noted improvement action, mentioned by respondents 
in 23 core sample schools, and the budgets also show that most schools (22) budgeted Year 1 SIG funds for 
professional development. However, there were some inconsistencies between SIG budgets and 
respondent reports. For example, respondents from 10 schools reported implementing new student 
behavior rules or programs at their schools, but the budgets revealed that only 3 schools planned to use 
SIG funds to fund these changes. This apparent discrepancy may be because many of the programs and 
strategies used by the core sample schools to improve student behavior did not have direct costs (e.g., 
new behavior policies could be enforced by school staff without buying any new software, hiring new staff, 
or incurring any other costs). More generally, discrepancies between site visit respondents and SIG 
budgets could be due to a variety of factors, including changed plans stemming from delays in SIG funding, 
incomplete or uneven knowledge across respondents about what SIG was funding, actions that were 
funded by SIG at the district level and thus not included in school-level budgets, and improvement actions 
described by respondents that were funded through non-SIG sources. For example, site visit data revealed 
that three schools hired staff midyear and one school was unable to hire the staff it had planned to hire 
during the 2010-11 school year due to SIG funding delays. Audited expenditure reports from 2010-11 will 
be analyzed in a future report to provide additional information on how SIG funds were actually spent on 
improvement actions in these core sample schools. 


28 The budget data show what schools intended to support with their SIG funds. Delays in receiving funds, 
additional restrictions set by districts overseeing the SIG allocations, and changes in school strategies may 
contribute to differences between these budgets and actual expenditures. Expenditure files were not yet available 
at the time of this report and will be the focus of a subsequent report. 
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Exhibit 5.2 

Average Estimated Year 1 SIG Per-Pupil Expenditures and the Range in the Percentage 
of Estimated Year 1 SIG Budgets for Core Sample Schools, 
by School Improvement Action 



Number of 
Schools 

Average Estimated 
Year 1 SIG Per-Pupil 
Expenditures 

Range in the Percentage 
of Estimated Year 1 SIG 
Budget 

Human Capital Management 




Professional development 

22 

$325 

1-41% 

Instructional coaches 

15 

$336 

3-27% 

Instructional leadership 

6 

$247 

<1-28% 

Teacher incentives 

6 

$186 

1-54% 

Technical Core 




Academic student supports 

19 

$427 

2-78% 

Curriculum and/or instructional changes 

19 

$387 

4-100% 

Technology (hardware and software) 

17 

$286 

<1-37% 

Nonacademic student supports 

13 

$380 

1-28% 

Targeted extended-time programs (after-school 
activities, Saturday school, summer school) 

12 

$260 

<1-80% 

Data use 

8 

$92 

<1-14% 

Extended day for all students 

7 

$371 

6-30% 

Early childhood education programs 

2 

$270 

5-35% 

Conditions That Support Teaching and Learning 




Parent activities 

10 

$121 

2-11% 

Strategies to change student behavior and/or 
increase safety 

3 

$76 

3-4% 

Indirect costs 

5 

$33 

1-3% 


Source: Year 1 (2010-11) SIG budgets provided by districts or from district SIG applications. 

Notes: Includes 25 core sample schools. Some expenditures are double counted if they fit into more than one category. For 
example, new classroom technology such as interactive whiteboards is included in both curriculum/instructional changes and 
technology. After-school tutoring services are included in both academic student supports and targeted extended- time 
programs. 


A Closer Look at Implementation: 

Three SIG-Required or SIG-Supported Actions 

We now more closely examine three improvement actions adopted by the core sample schools: teacher 
replacement, increased learning time, and student behavior policies/programs. The three actions differ 
in the degree to which their implementation was required or specified by SIG regulations. Each of the 
three improvement actions corresponds to one of the three domains in SST's conceptual framework and 
provide a glimpse at the role of SIG (explored in greater detail in Chapter 6) and the approaches to 
implementation in these schools. Teacher replacement, a human capital management action, was 
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required of schools implementing the turnaround model, and SIG specified the minimum proportion of 
teachers to be replaced (50 percent) and the timing of replacement (in Year 1). Increased learning time, 
a the technical core action, was required of schools implementing the transformation and turnaround 
models, but was only weakly specified in the SIG regulations. No particular amount of time was required 
and the additional time could be used for a variety of purposes. Finally, student behavior 
policies/programs were not required by SIG for any of the models (although they are permissible 
activities), yet prior studies have suggested that such interventions are often among the first changes to 
be instituted in low-performing schools seeking significant change (Herman et al., 2008). 

Teacher Replacement in Nine Core Sample Schools: 

Leveraging SIG Requirements to Address a Recognized Need 

Nine core sample schools replaced half or more of their teaching staff in the 2010-11 school year. 29 
Among these nine schools, six were implementing the turnaround model (which requires teacher 
replacement), two were implementing the restart model, and one was implementing the transformation 
model. The restart and transformation models did not require teacher replacement. 

Three core sample schools implementing the turnaround model did not replace teachers in 2010-11. 
One of these schools had replaced teachers before the 2009-10 school year and did not do so again in 
2010-11. The other two schools received SIG funds late in 2010-11 and thus did not replace teachers 
during that school year but planned to for the 2011-12 school year. 

The discussion that follows focuses on three aspects of teacher replacement in the nine schools that did 
so in 2010-11: the impetus for the teacher replacements, the teacher replacement process, and 
respondents' initial perceptions of teacher replacements. 

Impetus for a Teacher Replacement Strategy 

Most scholars and practitioners would agree that a team of skilled and motivated teachers is needed for 
a school to deliver quality instruction to students (Rice, 2003). The use of a teacher replacement 
strategy in low-performing schools reflects the perception that the central performance problem for 
some persistently low-performing schools is the quality of the teaching staff (Herman et al., 2008), and 
that changes in the composition of the staff are a prerequisite for meaningful change in teaching and 
learning. The descriptive literature indicates that schools that replace at least 50 percent of teaching 
staff may do so to meet a requirement (such as the for the SIG turnaround model), as a result of a 
principal or district decision, or because of voluntary teacher turnover, which is common in low- 
performing schools (Barnes, Crowe, & Schaefer, 2007). 

This section offers further detail on the impetus and context for teacher replacement in the nine core 
sample schools that replaced at least 50 percent of their teachers in SIG Year 1. We identified the 
impetus for teacher replacement based on reports from the district administrator and/or the principal 
(see Exhibit B.9 for more detail on the analytic procedures). Data from teachers provided additional 
details about teacher replacement. 

Respondents at seven of the nine schools that replaced teaching staff in the first year of SIG (six 
turnaround and one transformation) indicated that removals focused on staff members whom 
principals perceived to be less skilled or motivated. Principals and teachers described the replaced staff 


29 Of these nine schools, one replaced more than 50 percent of teaching staff in 2009-10 and then did so again in 
2010 - 11 . 
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members as "apathetic," "unmotivated," and "only doing the bare minimum," as well as "ineffective," 
"not up to the task," and "lacking credentials." For instance, the principal of one school commented that 
he asked staff to leave who were burnt out or set in an "old" way of doing things. 

Respondents at four of these six turnaround schools indicated that the schools were participating in 
non-SIG improvement initiatives that also required the replacement of at least 50 percent of the 
teaching staff. Respondents at these schools explained that plans to replace teachers had been under 
way prior to SIG. In some cases, the alignment of these precursor plans with the SIG requirement may 
have contributed to the selection of the turnaround model for these schools. Alternatively, schools 
implementing non-SIG initiatives may have anticipated the subsequent SIG grants and aligned their 
requirements accordingly. 

Even if SIG was not the sole impetus for teacher replacement in these schools, some respondents noted 
that the SIG program offered additional political support to make the teaching staff changes possible. 
This was the case in the two turnaround schools where SIG was the only reform initiative that required 
teacher replacement. At one of these two schools, the principal and a district administrator affirmed 
that they chose the turnaround model specifically because they had identified a need for stronger 
teachers, and SIG provided the political clout they needed to address this problem. 

The one transformation school is unique in that the principal spearheaded a teacher replacement 
effort in the absence of a SIG requirement to do so. He explained that he was seeking to build a staff 
that were committed to his new school improvement goals and qualified to carry them out. The 
principal had identified the need for a stronger staff when he began working at the school and 
commented that, "Coming to this school [prior to SIG], I studied data to see where pitfalls were." Similar 
to principals in the turnaround schools, this school leader argued that the teachers he replaced were 
"burnt out," "apathetic," or "lacking capacity." 

At the remaining two restart schools, teachers voluntarily left. Charter management organizations 
(CMOs) were slated to take over the schools prior to SIG, and while the CMO representatives 
encouraged teachers in the schools to apply for positions at the restart schools, none of the teachers 
elected to stay. Thus, all staff in these two schools were new in 2010-11. Neither CMO required the 
replacement of half of the staff but the SIG reform in conjunction with pre-SIG plans had led to complete 
staff turnover. 

The Teacher Replacement Process: Teacher Choice, Principal Authority, and Contextual Constraints 

The goal of teacher replacement is to remove teachers who are unmotivated, lack capacity, or are 
uncooperative, and replace them with teachers who are more highly motivated, skilled, and 
collaborative. The actual impact of the replacements, however, may be as much a product of the ways in 
which the process is implemented as of the qualities of the individuals involved. 

Descriptive studies of teacher replacement highlight three main processes: (1) removal or departure of 
teachers; (2) selection of interviewees from an applicant pool; and (3) hiring teachers. Each of these 
processes is governed by a combination of policies that include not only SIG requirements but also 
seniority rules and collectively-bargained agreements, as well as other requirements and initiatives 
(McMurrer, 2012; Scott & Kober, 2009). The extent to which teachers and principals have decision- 
making ability during these processes varies depending on the situation of the school. Principals may 
decide which teachers to remove, or teachers themselves may decide to leave (Scott & Kober, 2009). 
District officials may offer support to schools and principals and also may negotiate with union groups 
(Scott & Kober, 2009). Teachers may decide whether to apply to work at a low-performing school, or 
they may be transferred involuntarily (Calkins, Gruenther, & Belfiore, 2007). Principals may have 
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discretion about whom to hire from within their applicant pool, though teachers may or may not accept 
the position, and some principals may have difficulty recruiting qualified staff (Scott & Kober, 2009). 
Some principals may receive teachers who are transferred involuntarily to the school, without an 
opportunity to make the decision about hiring (Franck, Kelliher, & Varghese, 2011). 

A review of case study research suggested that successful turnaround schools had principals with the 
flexibility to select staff— which staff to remove as well as which staff to hire (Herman et al., 2008). In 
these cases, such flexibility was perceived by case study school respondents as essential to building a 
team that was on board with the principal's vision for how to improve the school. However, principals of 
low-performing schools are not always able to pull from strong applicant pools. One study that included 
a nationally-representative survey of districts reported that rural districts and districts with high-poverty 
students struggle to recruit qualified teachers (Birman et al., 2007). 

This section highlights two themes of the teacher replacement process in the nine core sample schools 
that used this strategy in 2010-11. (School classifications are based on reports from the district 
administrator and/or the principal. Data from teachers provided additional details about teacher 
replacement. See Exhibit B.10 for more detail on the analytic procedures.) First, most principals had 
discretion about which teachers would stay and which would leave the school. Second, at four schools, 
policies and contextual constraints limited the pool from which principals could select the replacement 
teachers. Both of these themes could have implications for the ultimate success of the teacher 
replacement efforts in these schools. 

Most principals at core sample schools that replaced teachers had discretion about which teachers to 
remove. In seven of the nine schools that replaced teachers, principals selected which teachers to 
remove. The principals who removed teachers described identifying staff members they felt were less 
skilled or less motivated. They explained that they identified teachers who were "burnt out" or "lacking 
capacity" and did not invite these teachers back to the school. One of these principals noted that the 
change might have been beneficial to the teachers and that they would be a better fit at a less- 
demanding, non-SIG school where they could perform at a "better pace." At the remaining two 
schools— both restart— CMO officials encouraged all teachers to apply to remain at the schools, but all 
the teachers chose to leave. Although the teachers who left these schools were not interviewed for this 
study, several current staff indicated that teachers decided to leave after the CMO takeover because the 
teachers preferred to remain in the district. Thus, the principals of these schools did not make the 
decision about which teachers to remove, but they were able to hire the entire group of teachers for the 
2010-11 school year. 

Four of the nine schools were in districts where teachers who were removed from turnaround schools 
were moved to other low-performing schools due to policies or other constraints on the pool of 
available teachers. Two of these schools, nested in a single district, selected teachers from an applicant 
pool that included teachers who had been removed or voluntarily transferred from other schools in the 
district. In this district, teachers were replaced in a two-step process. Teachers first had the opportunity 
to transfer voluntarily, and then the principal identified teachers he or she wanted to remove. Because 
there was a hiring freeze in the district, as well as a collectively-bargained agreement that tenured 
teachers could not be fired, the replaced teachers were transferred to other schools in the district. The 
superintendent noted: "Nobody was let go. It's one of the fallacies of the whole thing. People don't get 
fired; it was 'the lemon dance.'" In this district, teachers who opted to leave the schools and teachers 
who were removed from the schools were put on a districtwide "necessary transfer" list, which the 
district used to fill openings in other district schools, including other SIG schools. Another administrator 
from this district, who had chosen the turnaround model for one of the SIG schools, commented, 
"Unfortunately what we did here impacted the rest of the schools because we took 50 percent of the 
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staff, and we had to distribute those people to the rest of our schools." The principals of these two 
turnaround schools thus selected new staff members from an applicant pool that included teachers 
whom other schools had removed. 

At two schools in another district, all teachers were required to go through an application, interview, 
and observation process if they wished to keep their current position at the school. During this process, 
the building principals could avoid rehiring teachers they considered to be less skilled or unmotivated. 
Those teachers who were not hired after the application process were placed at another school within 
the district by "forced transfer." If these teachers were not happy with their placements, they could 
leave the district. One principal commented that it was "unfortunate" that teachers who had been 
"force transferred" were redistributed to other schools within the district. However, the principals and a 
district official did not report that these two core sample schools had received "forced transfers" during 
the first year of the grant. One of the principals commented that one of the things that made a "major 
difference" in SIG Year 1 was the ability to hire new and young staff who did not "have that defeated 
history." Site visit respondents noted that, in the past, this school had been a "dumping ground" for 
teachers whom other schools did not want. 

The Teacher Replacement Experience: Staff Perceptions 

Studies of accountability policies that result in replacing teachers indicate that the effort may yield 
benefits in terms of improved culture or instruction, but it also can be disruptive and induce tension 
(Allensworth, Ponisciak, & Mazzeo, 2009; Herman et al., 2008). We now report respondents' 
perceptions of the teacher replacement process after one year, recognizing that these perceptions may 
change over time. 

To understand respondents' initial perceptions of teacher replacement in the nine core sample schools 
that replaced teachers, we reviewed coded data from interviews with teachers, principals, and district 
officials on the subject of teacher replacement, and then classified each of the nine schools' perceptions 
of the teacher replacement as positive, neutral, or negative. Respondents' comments about teacher 
replacement could take into consideration perceptions of the quality of new teachers as well as 
perceptions of the teacher replacement process (i.e., the rules that govern adding and removing staff, 
together with the extent to which principals and teachers can make decisions about staff placements). 

In six of the nine schools that replaced staff in the first year of SIG (four turnaround and two restart 30 ), 
respondents characterized the addition of new teachers as positive for the school, bringing new 
energy and improved morale. In two others (both turnaround), respondents characterized the teacher 
replacement as neutral, just "another instance of change." For the remaining transformation school, 
respondents indicated that the teacher replacement process was demoralizing and aggravated 
problems for the school. Respondents at the positive schools felt that the new teachers brought needed 
energy and expertise to the building. For example, focus group teachers at one school expressed mild 
concern that the change was disruptive but felt that the addition of new teachers was a positive 
influence in the school overall. The new teachers raised expectations, brought expertise, and were 
regarded as strong and collaborative, according to the focus group. One teacher, who had been at the 
school for several years, said, "They are rising to the challenge, and they have high expectations. To me, 
it's like a rebirth." At another school in this group, all respondents (the principal, teachers, and 


30 At the two restart schools, all teachers, including the ones interviewed for this study, were new to the school. 
These teachers could not speak to the previous year or change from the previous year to the current one, but they 
were able to speak to their perceptions of the quality of the school's current staff. 
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instructional coaches) universally characterized the school's new staff as motivated and skilled, thanks in 
large part to the principal's decisions and leadership. These respondents indicated that the new 
teachers improved the school climate and instructional quality. 


Box 5.3. Perceptions of the Teacher Replacement Process 

School-level classifications on perceptions of the teacher replacement process are described below (see 
Exhibit B.ll for more detail on the analytic procedures). For this analysis, "respondents" come from the 
following respondent groups: district administrators, principals, and teachers. 

Positive 

• At least two respondents, regardless of respondent group, indicated that the new teachers 
introduced as part of the replacement process were beneficial (i.e., bringing new energy, 
improving staff morale, increasing teacher quality); AND 

• No respondent described the teacher replacement in terms such as biased or unfair. 

Neutral 

• Respondents did not comment on the quality of the teacher replacement process or the quality 
of new teachers, or described the process in neutral terms, such as another instance of change. 

Negative 

• At least two respondents, regardless of respondent group, indicated that the new teachers 
introduced as part of the replacement process were detrimental to the school (i.e., weakening 
staff morale, decreasing teacher quality); OR 

• At least two respondents, regardless of respondent group, described the teacher replacement 
process in terms such as biased or unfair. 

Both schools whose respondents expressed neutral sentiments with regard to the staff changes were 
located in the district where involuntary transfer teachers were moved to other schools in the district, 
including other turnaround schools. This is the same district in which the district administrator 
characterized the teacher replacement process as "the lemon dance." Focus group teachers also noted 
that there had been layoffs in the district in the past and that teachers were generally tired of changes in 
staffing. 

Respondents at the one negative school noted demoralizing experiences with the process of identifying 
teachers to be replaced. They explained that the principal "targeted" some teachers to be observed 
more than others, and they called the process unfair and intimidating. "There was a really unfair balance 
of observations," said one teacher. In addition, staff had mixed views about the resulting group dynamic 
among new and returning staff, although respondents did report that new Teach For America teachers 
who had been added were "bright and motivated." 

Teacher Replacement Summary 

Overall, the rationale for teacher replacement is to address a performance problem— the need for a 
more skilled, motivated, and collaborative staff. To this end, the principals of the turnaround and 
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transformation model schools 31 that replaced teachers during the first year of SIG all identified teachers 
for removal whom the principals perceived as less skilled or unmotivated. 

The relationship of SIG to the teacher replacement decisions is not straightforward, however. On the 
one hand, all turnaround schools either had complied or were planning to comply with the requirement 
that they replace at least 50 percent of the teaching staff. However, among the six turnaround model 
schools that actually replaced their staff in SIG Year 1, two were implementing SIG as their only initiative 
that required teacher replacement, while the other four schools were part of an additional non-SIG 
improvement program that also required teacher replacement. District administrators and principals 
from these four schools indicated that replacing at least 50 percent of teaching staff was planned to take 
place even in the absence of SIG and the turnaround model requirement, although in some cases SIG 
may have provided additional political clout to carry out this approach. 

The teacher replacement process— that is, the rules that govern adding and removing staff, together 
with the extent to which principals and teachers can make decisions about staff placements— matters in 
that it may influence the quality of the teachers who join the schools. Core sample schools each 
operated in a unique context of rules, requirements, and decision-making processes. The principals of 
the turnaround and transformation schools that replaced staff during 2010-11 all had the autonomy to 
decide which teachers would leave, suggesting that they could tweak their staff in a way that would 
align with their vision for improving the schools. However, at four of the turnaround schools, the 
teachers who were removed ended up in the applicant pool used to staff other schools in the district, 
including other SIG schools. Two of these turnaround schools were only able to add staff who had been 
removed from other schools (or had left other schools), and at the other two turnaround schools, some 
teachers who were removed were "forcibly" transferred to (or "dumped" on) other schools in the 
district, according to respondents. Mandatory school placements may not provide an appropriate fit for 
the individual teachers, and so such replacement processes may have implications for the receiving 
schools' abilities to build a staff that will meet their needs. 

Finally, a teacher replacement process may be seen as successful if it results in school staff who 
collectively are more skilled, motivated, and collaborative and who will remain at the school for 
sufficient time to have an impact on student learning. Although respondents at a majority of schools 
that replaced teachers in the first year of SIG expressed positive perceptions of the schools' new 
teachers, this was not the case in all schools. Moreover, it is impossible to know from one year of data 
how the changes in school staff will play out over time. Initial concerns about the process may dissipate 
as new and continuing staff learn to work together or as school personnel interpret changes in student 
outcomes. Conversely, new staff may become disillusioned or move on to other opportunities before 
substantial improvements in school functioning or performance can be realized. A third possibility is that 
the replacement of staff could be perceived as having the lasting and positive impact on the school that 
the policy designers envisioned. We will examine these issues in a future report. 

Increased Learning Time in 22 Core Sample Schools: 

A Case of Interpretation and Compliance 

In contrast to the straightforward requirement for turnaround schools to "rehire no more than 50 
percent" of the staff, SIG was less specific regarding increased learning time, which was required for 
turnaround and transformation schools but not for restart schools. These schools were required to 

31 At the remaining two restart schools that replaced teachers during the first year of SIG, all teachers voluntarily 
left. 
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"establish schedules and implement strategies that provide increased learning time" (U.S. Department 
of Education, 2010b), which was defined as: 

Using a longer school day, week, or year schedule to significantly increase the total number of 
school hours to include additional time for (a) instruction in core academic subjects..., (b) 
instruction in other subjects and enrichment activities that contribute to a well-rounded 
education..., and (c) teachers to collaborate, plan, and engage in professional development 
within and across grades and subjects. 

Advocates argue that spending more time in the classroom may increase student learning (National 
Center on Time and Learning, 2010). However, research on the effectiveness of increased learning time 
is mixed (Evans & Bechtel, 1997), and the extent to which extra time does influence student learning 
may depend on how that extra time is used (Kaplan & Chan, 2011). 

Site visit data indicate that 22 of the 25 core sample schools increased learning time during the 2010-11 
school year. Two additional schools did not implement any increase learning in 2010-11 but reported 
plans for more extensive increased learning approaches in 2011-12. These two schools also reported 
plans to increase learning time by adding an optional summer school offering in summer 2011. The 
remaining school in the core sample did not provide sufficient information to assess whether they 
implemented increased learning time. 

Impetus for Increased Learning Time 

SIG requirements, rather than an identified school need, appeared to be the main impetus for 
increasing learning time in the schools that did so. Of the 22 core sample schools that increased 
learning time in 2010-11, at least one respondent in 20 schools noted that this was a new action in the 
2010-11 school year. 32 (For this analysis, "respondents" come from the following respondent groups: 
district administrators, principals, teachers, instructional coaches, school improvement team members, 
parents, and students. See Exhibit B.12 for more detail on the analytic procedures.) According to 
respondents in 13 of these 20 schools, increased learning was implemented in part to comply with SIG 
requirements. The seven remaining schools, including three restart schools, reported increasing learning 
time in conjunction with district or CMO policies. 

Interpreting the Increased Learning Time Requirement 

Although most core sample schools (22 of 25) reported increasing learning time during the first year of 
SIG, the form that the increase took varied as a result of such factors as their interpretations of the 
requirement, the timing of when SIG funds were actually received, and the preferences of school 
leaders. By not specifying the amount of increased time and the students to whom it should apply, SIG 
granted schools considerable flexibility in how they could interpret and meet the requirement. Schools' 
implementation ranged from offering new or continued after-school learning opportunities for students 
to requiring students and returning teachers to stay an extra hour each day for formal instruction. 33 
Identified actions focused on the use of time are those reported by at least one respondent from the 
following respondent groups: district administrators, principals, teachers, instructional coaches, school 


32 Two schools that increased learning time in 2010-11 were excluded from this analysis due to insufficient 
qualitative data about why they increased learning time. 

33 The focus of our analysis here is on those increase learning opportunities involving students, and schools in 
which increased learning time consisted exclusively of professional development opportunities for staff are 
excluded. 
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improvement team members, parents, and students (see Exhibit B.13 for more detail on the analytic 
procedures). 

Core sample schools met the requirement by adding time onto the school day for all students, 
restructuring the way time was allocated during the school day, or offering optional increased 
learning opportunities. Nine core sample schools extended the school day by adding additional time to 
the regular school schedule, and three schools restructured the day to increase learning time within the 
existing school hours (see Exhibit 5.3). Most of the 22 core sample schools implemented optional 
increased learning opportunities for students (not shown), and for 10 of these schools, the optional 
opportunities were the only approach they used to increase learning time. These opportunities were 
offered through after-school programs or Saturday school. 


Exhibit 5.3. 

Number of Core Sample Schools Implementing Increased Learning Time in 2010-11, 
by Type of Activity 



Initiated Prior to 
2010-11 
(pre-SIG) 

Initiated During 
2010-11 
(Year 1 of SIG) 

Total 

Required increased learning time through an 
extended school day 

0 

9 

9 

Required increased learning time through a 
restructured school day 

0 

3 

3 

Offered optional opportunities as their only 
increased learning time 

5 

5 

10 


Source: SST respondent interviews and focus groups, spring 2011. 

Notes: Includes 22 core sample schools that implemented increased learning time in 2010-11. One of the nine schools that 
extended the school day during 2010-11 did not do so until March 2011. 


One factor that influenced the design of schools' increased learning time programs in 2010-11 was 
when schools actually received SIG funds. At six core sample schools, delays in receiving SIG funds 
reportedly affected the leadership's ability to implement its vision for extended day in the 2010-11 
school year. For example, the principal at one school, which received its funds in February 2011, noted 
that its 2010-11 implementation of an extended day was not ideal: 

The model [we adopted] took a six-period day and then added five minutes to each period to 
create thirty minutes of extended learning. You don't get much instructional traction out of five 
instructional minutes. That's not our goal, but that became a default solution because by the 
time we got the money. ..we had to take the path of least resistance while at the same time meet 
the provisions for the implementation of the grant. 

Several teachers at this school suggested that in the 2011-12 school year the additional time would be 
concentrated into a single additional period. Multiple respondents in two other core sample schools also 
reported modifying the structure of their extended days throughout the first year of implementation. 

In general, the timing of the SIG competition did not formally provide schools with a substantial 
implementation planning period prior to the first year of SIG in 2010-11. Respondents at two core 
sample schools indicated that they were thus using the 2010-11 school year to plan for an extended day 
to be implemented in the 2011-12 school year. For example, the principal at one of these schools 
indicated that although the school's after-school offerings increased learning time for some students in 
the first year of SIG, the school would be able to extend the day for all students in the second year due 
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to a policy they had crafted that allowed returning teachers to sign "elect to participate" forms 
acknowledging that they were working outside the union-negotiated teachers' contract. Busing 
arrangements would also be worked out for the students. 

Among the nine core sample schools (four elementary schools and five high schools) that 
implemented an extended school day program, the structure of the programs differed. School leaders 
determined how much time to add to the day and how students should spend that extra time. 

• Schools implementing an extended school day program for all students lengthened the school 

day by 2.5 to 5 hours per week, using the extra time to provide additional instruction for 
students and additional planning and professional development time for teachers. Four of these 
schools increased learning time for students on three to four days each week, and three of these 
four schools used the extra time on the other day(s) of the week for teachers to plan or 
participate in professional development activities. 

• The four elementary schools that extended their day used the extra time mainly to implement 
schoolwide or grade-level approaches to improving student achievement. In three of the four 
schools, the extra time was used for a schoolwide strategy to increase students' access to 
academic content or to focus on specific skills. At the fourth school, teachers in each grade level 
decided how to use the additional hour each day, whether it was, for example, for test 
preparation, science instruction, or visual arts. 

• Four of the five high schools that extended their day added a full class period. In two of these 
schools, students who needed additional support were assigned to academic interventions, and 
students who did not need additional support participated in electives or sports. The other two 
schools (located in the same district) combined academic and nonacademic enrichment. For 
example, a few days each week, students spent the additional period on academic activities, 
such as test preparation or senior projects or working on specific skills. In the remaining days, 
students spent the additional period participating in nonacademic enrichments, such as art and 
athletics. 

Perceptions of Increased Learning Time 

Although we are unable to assess whether schools and their teachers used increased learning time 
effectively, we can provide some insight on staff perceptions. The following discussion reports on the 
respondents' perceptions of increased learning time during the first year of SIG in 10 of 22 core sample 
schools that implemented increased learning time in 2010-11 (the remaining 12 schools were excluded 
due to insufficient qualitative data on perceptions). 

Respondents at five of the six schools with relevant data that extended the school day for all students 
expressed mixed opinions about the improvement strategy, and in the remaining school, respondents 
expressed consistently negative opinions. In the five mixed schools, teachers supported the extended 
day but critiqued implementation: "It had some glitches," "I would have liked to see more planning," and 
"We'd like to see [the extra time] used differently." In the one negative school, respondents said that 
adding five minutes to the end of each class period was unhelpful and only "overwhelmed" teachers. 
Although the school reportedly planned to restructure its extended day in 2011-12, teachers expressed 
concerns about adding an extra period without a clear sense of what was to be done during that time. 

Respondents at four schools with relevant data that implemented optional increased learning 
opportunities reported few challenges implementing their increased learning time. The principal at 
one of these schools noted that only two teachers thought that helping with after-school activities "is 
not my job" and that the rest had an "all hands on deck" attitude. At a second school, an external 
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support provider implemented after-school tutoring and supplemental learning, so the program fulfilled 
the SIG requirement for increased learning time without placing additional burden on teachers. The 
principal at a third school noted that teacher participation in the extended day program was a condition 
of working at the school and that “It is making everyone get on board that makes this place a different 
and special place." At the fourth school, a teacher noted the value of the small-group intervention 
during the school day that was made possible by the school's restructured day. 


Box 5.4. Perceptions of Increased Learning Time 

School classifications on perceptions of increased learning time are described below (see Exhibit B.14 for 
more detail on the analytic procedures). For this analysis, "respondents" come from the following 
respondent groups: district administrators, principals, teachers, instructional coaches, school 
improvement team members, parents, and students. 

Positive 

• At least two respondents described the implementation of increased learning time as beneficial 
to the school (i.e., enriches learning opportunities for students, provides students with extra 
support); AND 

• No respondent described increased learning time as detrimental to the school (i.e., extra time 
not used effectively, staff not supported in implementing increased learning time). 

Mixed 

• Respondents disagreed about the implementation of increased learning time. 

Negative 

• At least two respondents described the implementation of increased learning time as 
detrimental to the school (i.e., extra time not used effectively, staff not supported in 
implementing increased learning time); AND 

• No respondent described increased learning time as beneficial to the school (i.e., enriches 
learning opportunities for students, provides students with extra support). 


Increased Learning Time Summary 

SIG required schools implementing the turnaround and transformation models to increase their learning 
time, and our core sample schools complied with this requirement. These programs were initiated for 
reasons other than the SIG grant in 7 schools (including 3 restart schools), but the majority of schools 
(13) reported implementing increased learning time to comply with SIG. The vague stipulations of the 
law allowed SIG schools to interpret the requirement in a variety of ways. Schools implemented 
programs that ranged from adding a few additional minutes to each period to adding a new, hour-long 
period for all students. Other schools created additional time that was optional for students. For six 
schools, the timing of the grant affected how they implemented increased learning time in the 2010-11 
school year. Respondents' from schools that required increased learning time for all students tended to 
perceive the approach less positively, while respondents from schools that provided optional increased 
learning opportunities tended to be more positive (however, data on perceptions was unavailable for 
many of the core sample schools). In sum, extended learning time across the core sample schools during 
the first year of SIG was an example of compliance and interpretation. 
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Student Behavior Programs and Policies in 20 Core Sample Schools: 

Order as a Precondition for Learning 

SIG required some or all grantees to implement teacher replacement and increased learning time, 
although the degree of specificity for these actions varied. In the case of student supports, SIG provided 
very open-ended guidance: turnaround schools were "to provide appropriate social-emotional and 
community oriented services and supports for students," while transformation schools were permitted 
to "implement approaches to improve school climate and discipline" (U.S. Department of Education, 
2010b). Both models gave schools and districts considerable latitude as to how they chose to support 
students. Here we focus on one kind of support reported by core sample schools: programs aiming to 
improve student behavior. 

A safe and orderly school environment provides an essential foundation for students' academic success 
and healthy social-emotional development (Esposito, 1999; Haynes et al., 1997). Student misbehavior, 
including disruptive and violent behavior, is a major challenge for schools across the country, particularly 
those serving disadvantaged populations and those in urban areas (Robers et al., 2013). These problem 
behaviors create fear about personal safety and disrupt the learning environment, which can 
compromise students' ability to reach their full academic and social potential (Sprague & Walker, 2010). 

Respondents from at least three respondent groups in 10 core sample schools reported implementing 
new student behavior programs and policies in 2010-11 (see Exhibit 5.1). In an additional 10 schools, at 
least two respondents and/or the principal described student behavior programs and policies as part of 
the school's improvement actions. For the purposes of this section, we focus on the expanded set of 20 
core sample schools to more fully describe the range of reported programs and policies for student 
behavior. 34 

Impetus for Student Behavior Programs and Policies 

Respondents from 15 of the 20 core sample schools that were implementing strategies to improve 
student behavior during school hours identified student behavior as part of the school's performance 
problem. In these schools, individuals from at least one of the following respondent groups— at least 
two teachers, the principal, or a district administrator— described challenges associated with student 
behavior, including discipline problems, poor student attendance, behavior problems, a lack of student 
motivation, students with emotional issues, or weak student engagement (see Chapter 3 for a discussion 
of the perceived performance problems and Exhibit B.3 for more detail on the analytic procedures). 

Respondents at two schools felt that addressing behavior was the first step to improvement, something 
that needed to be in place before the focus could shift to instructional issues. For example, when asked 
what kind of support was needed at his school, a teacher union representative first mentioned behavior 
management and crisis intervention: "If you can't get through that," he said, "you can't teach in the 
classroom." A teacher at the other school made the same comment about his classroom, explaining that 
to meet academic goals, you "needed to get behavior in order." 


34 The analyses associated with student behavior programs and policies in this section were also conducted for 
only the 10 schools that met the original criteria of having respondents from at least three respondent groups 
describing student behavior programs and policies (see Exhibit 5.1). Results for these analyses shared the same 
patterns as the results for the larger set of 20 schools. 
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Common Student Behavior Programs and Policies 

Core sample schools adopted a range of programs and policies to improve student behavior, with 
most reporting that they implemented more than one. Identified student behavior programs and 
policies are classified as those reported by at least two individuals from the following respondent 
groups— district administrators, teachers, instructional coaches, school improvement team members, 
parents, and students— and/or the principal (see Exhibit B.15 for more detail regarding the analytic 
procedures). Although schools' approaches to addressing student behavior varied with each school's 
context, they did employ several common strategies for behavior management. Unlike increased 
learning time, which core sample schools implemented primarily to comply with SIG, schools chose to 
(but were not required to) implement programs and policies to address behavior problems. 

Respondents most frequently indicated that their schools had implemented the following strategies: 

• School uniforms. Seven core sample schools reported mandating school uniforms. At one 
school, several respondents spoke to how uniforms improved the culture and pride in the 
school. 

• Hiring additional staff to focus on student behavior management. Six core sample schools 
reported hiring new administrators or other support staff to focus solely or primarily on 
improving student behavior. The school leaders asserted that hiring staff or designating staff as 
disciplinarians was necessary to create a safe, respectful school community. For example, at one 
school, two deans and one assistant principal were designated as the school's "culture team." At 
another school, new climate specialists were hired in the 2010-11 school year. These staff 
members worked to create a common language and behavioral approach to poor behavior. The 
specialists also focused on instituting rewards for good behavior. 

• Positive Behavior Interventions and Supports (PBIS ). 35 Five core sample schools reported 
adopting the PBIS system. PBIS is a framework, rather than a specific curriculum, that guides 
decision making about research-based practices to improve student behavior and academic 
outcomes. PBIS programs are schoolwide and encourage prosocial behavior and behavioral 
expectations for students. PBIS systems focus on data-based decision making, embedded 
professional development, and coaching to promote schoolwide values and practices and thus 
establish a safe, social school culture (Horner et al., 2010). 

• Disciplinary room. Two core sample schools reported that there was a dedicated classroom 
space where students who misbehaved would be sent so that appropriate disciplinary action 
could be taken without losing instruction time. 

Perceptions of Student Behavior Programs and Policies 

Although there were some mixed or negative reactions, respondents' reactions to student behavior 
programs and policies were mostly positive across the 15 of 20 core sample schools that both reported 
implementing such programs/policies and provided sufficient qualitative data on their perceptions. 
Student behavior programs and policies themselves were varied and in varying stages of 
implementation. Respondents were not asked to comment explicitly on their "effectiveness," as 


PBIS, also referred to as School-Wide Positive Behavior Supports (SWPBS), is a widely-adopted approach to 
school behavior management and creation of a positive school environment. SWPBS is not a specific program or 
intervention but rather a framework for implementing and supporting evidence-based practices. For more 
information about PBIS, see the National Technical Assistance Center on Positive Behavioral Interventions and 
Supports at www.pbis.org. 
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assessing effectiveness is beyond the scope of this study, but respondents in 15 core sample schools did 
provide their perceptions on what they thought was working 'well' and what was not. 


Box 5.5. Perceptions of Student Behavior Programs and Policies 

School classifications on perceptions of student behavior programs and policies are described below 
(see Exhibit B.16 for more detail on the analytic procedures). For this analysis, "respondents" come from 
the following respondent groups: district administrators, principals, teachers, instructional coaches, 
school improvement team members, parents, and students. 

Positive 

• At least two respondents described the implementation of student behavior programs and 
policies as effective in managing student behavior; AND 

• No respondent described student behavior programs and policies as ineffective in managing 
student behavior. 

Mixed 

• Respondents disagreed about the effectiveness of student behavior programs and policies. 

Negative 

• At least two respondents described the implementation of student behavior programs and 
policies as ineffective in managing student behavior; AND 

• No respondent described student behavior programs and policies as effective in managing 
student behavior. 


In nine core sample schools, the new policies were collectively perceived by at least two respondents to 
be working well to control student behavior. In four schools, teachers reported that the policy or 
program was either not working or had not been effectively implemented. For example, a teacher in one 
school reported that teachers and school administrators "were not on the same page" and when she 
sent students to the principal's office to address a discipline concern, the principal simply sent the 
students right back to her. In the remaining two schools, respondents provided mixed reactions. For 
example, respondents had mixed evaluations of the creation of new classrooms where students who 
misbehaved were sent. At one school with a newly-established discipline room, teachers reported being 
grateful to have a place to send disruptive students, but some teachers were not sure whether this 
strategy actually improved behavior in the classroom. 

Student Behavior Summary 

In contrast to requirements for teacher replacement or increased learning time, SIG does not require 
schools to implement policies and programs to specifically improve student behavior. Yet 80 percent of 
the core sample schools reported doing so. In most cases (15 schools), these programs and policies 
responded directly to a staff perception that poor student behavior was an important part of the 
school's performance problem. The lack of specificity in the law allowed schools to decide both whether 
and how to implement such programs and policies. Schools did in fact report putting in place a broad 
range of approaches, from school uniforms to dedicated staff and disciplinary structures to schoolwide 
programmatic frameworks such as PBIS. Staff reactions to these policies were mostly positive, although 
there were also some mixed or negative reactions. 
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Chapter Summary 

This chapter has examined the number and type of improvement actions implemented across the 25 
core sample schools. In accordance with SIG requirements, the most commonly-reported improvement 
strategies were increasing professional development time, replacing principals, and increasing learning 
time. Although many identified actions were reportedly supported with SIG funds, others were not. 

Overall, respondents at core sample schools identified 11 improvement actions during the 2010-11 
school year. These 11 actions each fall into one of the three domains outlined in SST's conceptual 
framework: human capital management, the technical core, and the conditions that support teaching 
and learning. All the improvement actions appeared in schools regardless of the particular SIG 
intervention model that was being implemented. 

This chapter also further described the implementation of three particular improvement actions: 
teacher replacement, increased learning time, and student behavior programs and policies. The purpose 
was to begin to illuminate differences in the use and implementation of these actions, which varied in 
terms of the degree to which SIG required and specified the actions. 

The rationale for implementing each of these three actions differed. Despite the SIG requirements for 
teacher replacement being very specific, respondents indicated that the rationale for teacher 
replacement was less often about being compliant and more often about needing to address a 
perceived performance problem: the lack of a sufficiently skilled, motivated, and collaborative staff. 
Schools and districts may have even selected the turnaround model because they already planned or 
wanted to replace staff, and SIG simply provided additional authority and political clout to do so. In 
contrast, respondents indicated that they implemented increased learning time programs primarily to 
meet SIG requirements. Finally, although student behavior programs and policies were not specifically 
required by SIG, many core sample schools still implemented them. They appeared to do so primarily 
because they perceived poor student behavior as contributing to poor school performance. Thus, SIG 
requirements did not appear to influence the decision to enact student behavior policies and programs, 
although the extra funding may have helped facilitate their enactment. 

The various improvement actions taken in the first year of SIG illustrate the core sample schools' self- 
identified priorities to improve student learning, as well as the priorities for SIG schools set forth in the 
federal grant's rules. Both the specific actions and the priorities among them may change over time as 
implementation unfolds and as schools near the end of grant funding. 
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Chapter 6: SIG and the Change Process 


SIG intends to catalyze dramatic change in the nation's persistently low-performing schools— many of 
which had engaged in multiple improvement efforts prior to SIG without achieving sufficiently deep or 
sustained results to substantially alter their performance trajectories. The actions of the core sample 
schools reported in Chapter 5 suggest continued or renewed effort to improve. However, simply 
cataloguing and counting the improvement actions undertaken by these schools will not tell us whether 
the schools are producing a change process that will yield lasting improvement. The purpose of this 
chapter is to examine SIG-inspired or related actions in light of what existing research suggests about 
the process of school change, including how change is initiated, supported, and constrained. 

Unfortunately, the research base on school change is far from definitive. For decades, school reformers 
have described specific school improvement strategies related to human resources, the technical core of 
instruction, conditions that support learning (the three domains described in our conceptual 
framework), or combinations of these strategies situated within more comprehensive whole-school 
reforms. In the 1980s, case study research on "effective schools" identified characteristics of schools 
that had "beaten the odds"— that is, schools whose students were performing better than would be 
expected given the attributes of their students (Edmonds, 1979; Levine & Lezotte, 1995; Purkey & Smith, 
1983). Yet these studies say little about how schools could actually change from low to high 
performance. In the 1990s and early 2000s, researchers and policymakers turned their attention to 
comprehensive school reform (CSR) models aimed at providing whole-school interventions to 
reengineer and redesign all aspects of school operations (Aladjem et al., 2006). These studies identified 
challenges in implementing CSR (Aladjem et al., 2006; Herman et al., 1999) and produced scant evidence 
of lasting success. Even when school reforms have achieved positive results, case studies suggest that 
school improvement has often taken three to five years— too slow for schools to meet the demands of 
an impatient policy audience or the performance targets set by state and federal accountability policies 
(Aladjem et al., 2010). 

The recent focus on school turnaround seeks to move both policy and practice past these slower, more 
incremental approaches, suggesting that what is needed are more dramatic interventions that signal a 
departure from prior practice (Herman et al., 2008). Whether the kind of disruption posited in the 
recent literature on school turnaround (and reflected in SIG) actually produces a more rapid and 
sustained change process, however, has not been empirically demonstrated. One goal of this study is to 
explore the hypothesized link between disruption from the past and sustained change in school 
practices in our core sample schools. To that end, this chapter focuses on the extent to which visible 
departures from past practice (some of which are required by the SIG models) took place in the core 
sample schools by the end of the first year of implementation. Subsequent data from the second and 
third years will enable us to examine the role of disruption in stimulating a sustained change process. 

This chapter analyzes data on the actions undertaken in the 25 core sample schools during the 2009-10 
and 2010-11 school years (the prior and first years of SIG). Based on site visit and teacher survey data, 
we first examine whether the actions in these schools signaled a disruption from the past and a dramatic 
new direction for the school. We then turn to the role of SIG in the school change process, taking into 
account the observation that in many core sample schools, specific improvement actions had been 
initiated prior to the school receiving SIG. We then examine aspects of SIG program implementation 
during the first year of the grant. In this discussion, we pay particular attention to the level of 
stakeholder participation in the SIG application process, as well as the timing of the funding. Finally, we 
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look at the external supports that schools reported receiving, which taken together may have 
contributed to or impeded the role of SIG in the change process. 


Box 6.1. Key Chapter 6 Findings 

• SIG was designed to initiate actions that would be perceived as dramatic, visible changes, and 
respondents in 7 of the 25 core sample schools described a set of activities (such as replacing 
half of the teachers, or a change in school governance) that together constituted a disruption 
from past practice. 

• In 19 of the 25 core sample schools, SIG was not perceived to be the primary impetus for 
change, whether or not the schools had experienced disruptions from the past. In these 19 
schools, SIG was incorporated into a reform process that had been planned or launched the year 
prior to SIG, in 2009-10. In another four schools, SIG was perceived as the primary impetus for 
change, while in the remaining two schools, the first year of SIG reflected "business as usual." 

• The process of applying for SIG and delays in the receipt of funds posed challenges to early 
implementation of the grant. In 10 of the core sample schools, school stakeholder involvement 
in the SIG application process was reportedly limited. In six schools, there was reportedly no 
school stakeholder involvement. In addition, nearly half of the core sample schools (12 of 25) 
reported experiencing delays in the implementation of specific improvement actions because 
they did not receive actual SIG funds until after the start of the 2010-11 school year. 

• Although 20 core sample schools reported receiving compliance-focused monitoring and 
guidance, just 10 reported receiving support for their improvement efforts. In general, districts 
reported providing support for school improvement more often than core sample schools 
reported receiving such support. 


Disruption From the Past 

Grounded in literature from business management and informed by case studies of schools that have 
rapidly "turned around" their students' performance (i.e., in three years or less), researchers and 
policymakers have begun to identify factors that seem to be present in schools that turn around 
(Aladjem et al., 2010; Hassel & Hassel, 2009). In particular, retrospective case studies suggest that low- 
performing schools that have improved their performance appreciably over a short timeframe made 
dramatic changes from the status quo by signaling the urgent need for change, making visible 
improvements right away ("quick wins"), sharply focusing on instruction, and building a committed 
staff— often through releasing, replacing or redeploying school personnel (Bryk, et al., 2010; Herman 
et al., 2008). 

Interpreting these findings, turnaround advocates and researchers argue that to effectively address 
long-standing, intransigent patterns of low performance requires a dynamic, intensive, sustained change 
process that starts with a visible disruption of existing school norms and practices (Hassel & Hassel, 
2009). The pivot point for this process may be demarcated through a set of visible actions designed to 
overcome inertia, challenge assumptions, and signal the end to business as usual. Individual actions 
could be primarily symbolic (such as changing the name of the school) or substantive (such as replacing 
staff or adding instructional time to the school day for all students), but taken together, they are 
believed to create a recognizable departure from prior practices, norms, and assumptions. Despite the 
limited research base, advocates assume that this departure is necessary to initiate a focused and rapid 
improvement process. 
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Reflecting this assumption, SIG requires or encourages a number of visible changes, including 
replacement of the principal (turnaround and transformation models), replacement of at least 50 
percent of the teachers (turnaround), or governance change (restart). To these actions, school leaders in 
the core sample schools added other visible changes that in some instances combined to sufficiently 
signal a break with past practice. In this section, we examine whether respondents describe the 
following eight potentially disruptive actions 36 as "visible changes" and whether these changes, taken 
together, were perceived as a disruption from the past: 

• Replacement of the principal. The school or district hired a new principal. 

• Replacement of at least 50 percent of teachers. At least 50 percent of teachers were new to the 
school at the start of the school year. 

• Governance change. An education management organization or charter management 
organization took over the school. 

• Mandatory extension of the school day. The school extended the length of the day for all 
students. 

• Changes to school organization. This includes splitting the school into small learning 
communities (SLCs), dividing the school into multiple smaller schools, combining the school with 
another school or school(s), or adding grades to the school. 

• Changes to the physical plant of the school. The school made physical updates to the school, 
such as rebuilding a school structure, repairing dysfunctional equipment (e.g. bathroom, 
cafeteria), or making aesthetic improvements (e.g., schoolwide painting). 

• Public communication about school changes. School changes were communicated to the public 
through such means as school media appearances (T.V., newspaper) and door-to-door 
campaigns aiming to disseminate information about the school. 

• Other visible changes. The school experienced other visible acts that do not fall into one of the 
above categories. For example, the principal of one school explained that he changed the name 
of the school to signal to the community that it was not the place it was before and that it was 
now a place with rigorous instruction and high expectations. 

In carrying out these analyses, we did not assume that such a disruption will necessarily lead to desired 
or sustained improvements in student performance. Some schools that experience disruption may not 
continue with the kinds of classroom-level reforms to teaching and learning that have been tied to 
student outcomes. Furthermore, actions that are not reflected in the school turnaround literature and 
SIG may in some cases stimulate sustained improvement activity without such a visible break. Our 
analysis of whether schools appeared to experience a disruption from the past covers the prior (2009- 
10) and first (2010-11) years of SIG. 37 


36 These "potentially-disruptive actions" were derived from the literature on "quick wins" and symbolic changes 
(Hassel & Hassel, 2009; Herman et al., 2008), SIG regulations, and descriptions of disruptive actions from our core 
sample school respondents. 

37 Interview and focus group data suggest that two schools in the core sample underwent a "disruption from the 
past" several years before SIG. One was divided into smaller learning communities following a period of social 
unrest in the community, and the other had been reconstituted in the early 2000s. Due to concerns about 
respondent recall and turnover among staff, we excluded these events from the analysis and did not count them 
as being a disruption from the past. 
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Box 6.2. Visible Changes and Disruption From the Past 

School classifications on disruptions from the past are described below (see Exhibit B.17 for more detail 

on the analytic procedures). 

Experienced a disruption from the past 

• The school demonstrated a visible change for at least four of the eight indicators described 
above in either 2009-10 or 2010-11. (Indicators demonstrating a visible change are those which 
analysts determined were of sufficient magnitude based on the preponderance of evidence 
from the following respondent groups: district administrators, principals, teachers, and 
instructional coaches.) The use of four indicators as the threshold was informed primarily by a 
"natural" break in the data. 

Did not experience a disruption from the past 

• The school demonstrated a visible change for less than four of the eight indicators described 
above. 


Respondents from seven core sample schools described four or more visible changes, so these schools 
were classified as having experienced a disruption from the past. Six of these schools underwent a 
disruption in 2010-11, the first year of SIG implementation, and one school underwent a disruption in 
2009-10, the year prior to SIG implementation (see Exhibit 6.1). For example, in 2010-11, respondents 
from one school reported the following visible changes: (1) replacing the principal; (2) replacing all 
teachers; (3) allowing a CMO to take over the school's governance; (4) repairing the cafeteria to signal a 
change in culture; (5) lengthening the school day and school year for all students. The school leaders 
also implemented new rules, routines, and procedures such that a teacher described the experience as 
"culture shock." The school instituted mandatory uniforms, and the summer before the school reopened 
as a charter, the new principal and assistant principal went door-to-door introducing the school to 
parents and alerting them to changes that would arise. 

The one school that appeared to experience a disruption in the year prior to SIG had five visible changes: 
(1) hiring a new principal, assistant principal, and instructional coach; (2) replacing more than 50 percent 
of staff; (3) renaming the school to signal a new beginning; (4) lengthening the school day; and (5) 
broadcasting the changes publicly to the community. As one coach explained, with the new "visionary" 
principal, the school was "in the media for good reasons... now it's a culture of empowerment and 
vision." 

Other schools in the core sample experienced fewer visible changes, the most common being a new 
principal. However, if these new principals did not implement other visible actions, the school was not 
rated as having experienced a disruption from the past. For example, in 2010-11, one school had a new 
principal who sought to change teacher expectations for student performance. However, teachers were 
not replaced, the physical plant of the school was not changed, organization or governance were 
unchanged, and there were no other symbolic acts to communicate a departure from business-as-usual. 
The school thus did not have enough visible changes in 2010-11 to be considered as having experienced a 
disruption from the past. 
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Exhibit 6.1. 

Number of Visible Changes Among Core Sample Schools, 2009-10 and 2010-11 


Baltimore Bridge Elementary 
Sawbuck Elementary 
Sterling Slope Elementary 
West Marble High 
Big Acorn High 
Inner Brooks High 
Meribel High 
Blizzard Bay Elementary 
Island Bay Elementary 
Peregrine Hill Elementary 
Proctor Point High 
Raven Ridge Elementary 
Tyro Trail Elementary 
Melon Elementary 
Haven Way Elementary 
Elmsville High 
Gillepsie High 
McAlliston High 
Rossignol Elementary 
Sherbrooke Elementary 
Aerovista High 
Coral High 
Tyron Elementary 
Gale Secondary 
Paul Bunyan High 

Number of Visible Changes from the Past 


Source: SST respondent interviews and focus groups, spring 2011. 

Notes: Includes 25 core sample schools. All school names are pseudonyms. Totals per school reflect visible changes in either 
2009-10 or 2010-11, whichever was greater. 

Of the seven schools in the core sample that experienced a disruption from the past, one was 
implementing the transformation model. This school replaced more than half of its teachers, even 
though SIG did not require it to do so. The other six schools that experienced a disruption from the past 
were implementing either the turnaround (four schools) or restart (two schools) models. This pattern 
may not be surprising, given the requirements for adopting these models. However, even though the 
timing of these schools' disruption from the past coincided with their adoption of a SIG model, the 
activities that constituted their disruption were not necessarily attributable to SIG. We next discuss the 
extent to which SIG was perceived by respondents to be the primary impetus for a change process that 
may have included a disruption from the past. 

Perceived Role of SIG in the Change Process 

Case study research has documented how schools' contexts and experience with prior improvement 
efforts may influence how new initiatives are perceived and implemented (Scott, 2009). As described in 
Chapter 3, respondents in five core sample schools reported a history of neglect by their districts while, 
in contrast, three core sample schools reported being the subject of a great deal of attention from the 
district for decades. As discussed in Chapter 5, our core sample schools often had commenced specific 
improvement actions prior to receiving SIG. As discussed earlier in this chapter, one core sample school 
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had even experienced a disruption from the past in the year prior to SIG. Finally, 10 core sample schools 
were located in districts that had launched turnaround policies of their own just prior to SIG. These 
policies entailed activities similar to those required under some SIG models, including changing 
governance structures, replacing school leaders, and recruiting new teachers. As a result of this diversity 
in history and context, we hypothesize that the contribution of SIG is likely to be perceived differently in 
schools, depending on their particular history and context. 

Here we focus on the role of SIG as a perceived catalyst for the school change process, and how this 
perception is related to prior improvement efforts. Across the core sample schools, SIG was perceived to 
fit into the change process in two different ways: as the primary impetus of change or as an effort that 
fit into an ongoing change process. For a third group of schools, interview and focus group data 
suggested that the change process had not yet begun, and the schools were experiencing business-as- 
usual (see Box 6.3). 


Box 6.3. Centrality of SIG in the Change Process 

School classifications on the centrality of SIG in the change process are described below (see Exhibit B.18 
for more detail on the analytic procedures). For this analysis, "respondents" refers to the principal and 
at least two other respondents from the following respondent groups: district administrators, teachers, 
and instructional coaches. 

SIG catalyzed change that took place in 2010-11 

• Respondents described SIG funding and requirements as the primary impetus for change; that 
is, changes would not have happened were it not for the grant. For instance, SIG may require 
the replacement of a school's principal or staff (and then the new principal or new staff may 
drive change at the school). 

SIG fit into an ongoing change process 

• Respondents described organizations or policies (e.g., district reforms) outside of SIG as the 
primary impetus for change, although the grant may have augmented or deepened ongoing 
reforms. For schools in this category, either a change process was launched prior to Year 1 of 
SIG, OR a change process was planned prior to SIG and launched concurrent with Year 1 of SIG. 

SIG supported business-as-usual 

• Respondents described 2010-11 changes as "marginal" or "tweaks," indicating that although 
the schools had made purchases with SIG funding, they had not launched a reform process. 

Respondents in four core sample schools (three turnaround and one transformation) characterized 
SIG as the primary impetus of the change process experienced in 2010-11. This change was viewed as 
a positive experience in some schools but not in others. In the three turnaround schools, school 
principals and district administrators viewed SIG as affording the necessary leverage to replace 
ineffective teachers, who were perceived as contributing to the school's performance problem. 

At one of these turnaround schools, half of the teaching staff was replaced in 2010-11 as required by 
the model, a change that respondents described as an overall positive influence in the school. According 
to administrators and other respondents, the school was a "dumping ground" for the district's 
ineffective teachers in the past, and in recent years, its teachers were not properly credentialed. 
According to teacher respondents, the new teachers raised expectations, brought expertise, and were 
regarded as strong and collaborative. 
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At the two other turnaround schools, although SIG was described as the catalyst for change in 2010-11, 
school respondents perceived these changes negatively. Teachers at one of these schools indicated that 
the principal switched teacher grade-level assignments to create an uncomfortable working 
environment in the hopes of getting half the teaching staff to leave voluntarily. The principal confirmed 
this account, explaining that the move was done at the instruction of the district "so, by default, 
[teachers] would leave and [administrators] wouldn't have to have those uncomfortable conversations 
with the union or anything." At the other school, respondents indicated that the prior principal, 
although well regarded, was removed because of SIG requirements and was replaced by a new principal, 
whom the majority of respondents (including teachers, both instructional coaches, the union 
representative, and one parent) viewed unfavorably. Many described him as "hands off," disorganized, 
and not fully committed to improving the school. As one teacher explained, "Things were done based on 
an agenda. ..they were not done necessarily for the benefit of the school." 

At the fourth school, the transformation model was viewed as the only viable choice. According to one 
district administrator, closing the school was not a feasible option in the district, the school board was 
not interested in charter schools, and due to the district's strong union presence, replacing a majority of 
teachers was untenable. Respondents credited the new principal, hired in 2010-11 as required by the 
transformation model, with introducing a culture of high student expectations. According to several 
school respondents, every morning the principal would greet students as they arrive to school, shouting 
words of encouragement such as, "You gotta get to class and you gotta learn to read so you can go to 
college!" At the start of the school year, the principal met individually with every student in grades 4-6 
to discuss their career and college aspirations and then posted their responses alongside their pictures 
in the hallways. 

Respondents in 19 core sample schools (10 transformation, 6 turnaround, and 3 restart) described SIG 
as fitting into and intensifying an ongoing change process but not as the primary impetus for change. 

Respondents from 12 of these 19 schools reported that a change process was initiated prior to the first 
year of SIG, while respondents in the remaining 7 schools reported beginning reforms concurrent with 
SIG. Respondents at these 19 schools perceived SIG as supporting or building on existing district or 
school reform efforts, indicating that the selection of SIG intervention models was driven primarily by 
the model's fit with ongoing initiatives. For example the district administrator for one school suggested 
that the transformation model included many activities that were either already in progress at or 
planned for the school, such that "everything just slid into that model." Likewise, respondents in two 
other schools indicated that the transformation model was most aligned with the actions proposed in 
their respective restructuring plans, and the three restart schools were assigned to undergo CMO 
takeover in 2010-11 as part of a district reform initiative. 

Two schools had applied to participate in a new district-led reform initiative, which, like the SIG 
turnaround model, required the replacement of the principal and 50 percent of teachers, among other 
changes. According to a district administrator, the alignment of the district turnaround model with the 
SIG turnaround model was deliberate, but the district selected schools for this program prior to SIG, and 
decisions were not contingent on SIG funding, as the schools would have proceeded with these changes 
regardless of whether the district's SIG application was accepted by the state or not. However, the 
district administrator also reported that the changes in these two schools were deepened with SIG 
resources. For example, one school used 2010-11 SIG funds to pay for more instructional coaches and 
to improve school facilities and grounds (the school added a new playground, a new garden, and a 
fence), among other improvements. The principal explained that the SIG funding "allowed me to 
transform the school in a manner that was quick and concise" such that "we see results immediately." "I 
think we have everything we need," he said. 
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Respondents in the remaining two core sample schools reported changes in 2010-11 that were 
limited in scope, suggesting that neither school had embarked upon a reform process and that SIG 
funding was supporting business-as-usual. These two schools, which were located in the same district, 
reported improvement actions that largely consisted of the purchase of technology-related resources 
and reportedly did not alter instructional practices (i.e., technology either was not integrated into the 
curriculum, or it was used to support traditional drill and practice). Compared with other schools in the 
core sample, these schools received grant amounts that were average or above average, so the 
apparent lack of reform activity does not appear to be attributable to the SIG funding levels. Moreover, 
the district's SIG application proposed continued use of a school reform model that had been 
operational in the district since 1997. As indicated by district officials, the district selected the 
transformation model for its SIG schools because they were not ready for something more disruptive. 
Thus, despite SIG funding and requirements, these two schools had not yet launched a new reform 
process. Indeed, as the principal of one of these schools observed, "Our school has the transformation 
model, but nothing has been transformed." 

Of the seven core sample schools that experienced a disruption from the past, SIG was perceived to 
be the impetus for this disruption in just one school, while in the other six schools, SIG was perceived 
as fitting into an ongoing change process (see Exhibit 6.2). At the one school where SIG was perceived 
to be the impetus for change, the principal and district administrator indicated that SIG, or more 
specifically, the SIG turnaround model, provided the necessary leverage to replace the teaching staff and 
as a result, the school replaced more than 50 percent of its staff in the first year of SIG. 

At the other six schools, respondents described organizations or policies other than SIG as the primary 
impetus for the disruption from the past. These schools either launched a change process prior to SIG in 
2009-10 (one school) or concurrently with SIG in 2010-11 (five schools). Among the latter set, one 
restart school, for example, was turned over to a CMO by the district as part of a non-SIG reform 
initiative. Other indicators of visible change evident at this school included a new building, a longer 
school day, and mandatory school uniforms. Respondents at this school did not ascribe these changes to 
SIG and, in fact, were unaware of the program. None of the teachers in the focus groups had heard of 
SIG, and even the principal was not cognizant that the school had a grant. In other words, for these 
schools that experienced a disruption from the past, SIG appeared to fit into a change process that had 
been planned or launched prior to SIG, although SIG funds did support this previously-planned change 
process. 

Of the 18 core sample schools that did not experience a disruption from the past, respondents in 13 of 
them reported that SIG enhanced or augmented an existing change process. One of these schools 
began the reform process in the year prior to SIG, when a new superintendent and a new principal were 
hired. These two new leaders worked closely together, planning for changes to address the school's 
changing demographics, increased behavior problems, stagnant teaching strategies, and slipping test 
scores. The superintendent conducted a needs assessment to understand what teachers, parents, and 
students thought of the school, and this process revealed concerns over safety and teaching quality. In 
that first year under new leadership, an external organization provided professional development to all 
elementary teachers in the district, focusing on pedagogy and delivering curriculum in the classroom. 

The principal at this school cracked down on student behavior problems, logging the highest suspension 
rate in the school's history, until new behavior norms were accepted. The superintendent said, "[The 
principal] got the gold star for suspensions that month, [but] she's not number 1 in suspensions this 
year; it helped transform the culture." When the opportunity to apply for SIG arose, the superintendent 
said it aligned cleanly with the reform process they had already begun. After extensive stakeholder 
involvement in the planning process, SIG paid for classroom technology to further invigorate teaching 
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strategies, a contract for another external provider the principal knew from a previous district, and 
additional professional development hours for teachers, along with additional programs and staff to 
improve student behavior. In this way, administrators were able to ensure the SIG-funded improvement 
actions both augmented and were aligned with activities that were initiated prior to SIG. Although this 
school experienced several visible changes, the number of such changes in a single school year did not 
meet our threshold to be considered as experiencing a disruption from the past. Indeed, over the two 
years described, changes at this school were sequenced and cumulative, rather than disruptive. 


Exhibit 6.2. 

Centrality of SIG in Change Process in Core Sample Schools, 
by Reports of a Disruption From the Past 



SIG Perceived as Primary 
Impetus of the Change 
Process 

SIG Perceived to Fit Into an 
Ongoing Change Process 

No Change Process 
(Business-as-Usual) 

Total 

Experienced 
Disruption 
From the 
Past 

1 school 
Meribel High 

6 schools 

Baltimore Bridge Elementary 
Big Acorn High 
Inner Brooks High 
Sawbuck Elementary 
Sterling Slope Elementary 
West Marble High 


7 

schools 

Did Not 
Experience 
Disruption 
From the 
Past 

3 schools 

Blizzard Bay Elementary 
Island Bay Elementary 
Tyron Elementary 

13 schools 
Aerovista High 
Coral High 
Elmsville High 
Gillepsie High 
Haven Way Elementary 
Melon Elementary 
McAlliston High 
Peregrine Hill Elementary 
Proctor Point High 
Raven Ridge Elementary 
Rossignol Elementary 
Sherbrooke Elementary 
Tyro Trail Elementary 

2 schools 
Gale Secondary 
Paul Bunyan High 

18 

schools 

Total 

4 schools 

19 schools 

2 schools 



Source: SST respondent interviews and focus groups, spring 2011. 

Notes: Includes 25 core sample schools. All school names are pseudonyms. 


Initial Administration of SIG 

Whether SIG is perceived as a catalyst for new change or as a supplement to previously-planned change, 
we hypothesize that the initial administration of SIG and the initial technical support for implementation 
from states, districts, and external providers also may influence the grant's role in the change process. 
This section reviews two aspects of the initial administration of SIG: school-level involvement in the SIG 
application process and the timeliness of SIG funding, both of which could shape initial school-level 
impressions of the grant. Then in the next section, we describe the initial support provided to schools by 
states, districts, and external partners, as perceived by the respondents in our core sample schools. 
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Involvement in the Application Process 

As part of the competitive subgrant process held within each state that received a formula-based share 
of federal SIG funds, districts were responsible for applying for SIG funds on behalf of their SIG-eligible 
schools that they committed to serve. Specific application requirements varied by state, but they 
generally required districts to: (1) demonstrate commitment to and capacity for turning around the 
school(s); (2) identify the improvement model for each school; (3) consult with relevant stakeholders 
regarding the selection and implementation of the improvement model; and (4) include a budget for the 
use of SIG funds (see Hurlburt et al. [2011] for further details on state requirements for their districts' 
SIG applications). According to federal SIG guidelines, districts were expected to "consult with relevant 
stakeholders, as appropriate, regarding [the district's] application and implementation of [the] school 
improvement model in Tier I and Tier II schools" (U.S. Department of Education, 2012, p. 56). In addition, 
the guidelines "strongly encourages" the inclusion of students' families and community members in the 
decision regarding the selection of an improvement model and strategies. 

There is a long-standing assumption among policymakers— and reflected in early correlational studies of 
comprehensive school reform— that involving school-level stakeholders in the selection of school 
reforms can improve implementation of the reform in the long run (Bodilly, 1998; Datnow, 2000; Weiss 
& Cambone, 1994). This may be because greater involvement of school-level staff can facilitate a better 
fit between reform strategies and the perceived performance problems in the school, and greater 
involvement and understanding of the purposes of the grant can facilitate stakeholder buy-in. However, 
other studies call into question whether stakeholder involvement in the early planning process of 
reform is necessary or desirable. For example, one study suggests that changing behavior can lead to 
subsequent changes in beliefs about the reform process, even in the absence of participation in the 
decision-making process (Fishbein & Ajzen, 1975). One other mixed-method study found that 
stakeholder involvement in the process of adopting a comprehensive school reform model did not 
necessarily ensure greater fidelity of implementation (Le Floch, Zhang, Kurki, & Hermann, 2006). 

With regard to SIG, such early involvement of relevant stakeholders faces additional complications. For 
example, given the assumptions implicit in SIG with regard to a visible break from the past, engaging 
stakeholders (who may be associated with those entrenched practices) may be counterproductive. 
Moreover, SIG requirements to replace principals (in three models) and teachers (in one model) raise 
questions about the potential benefit of stakeholder involvement, given that staff who implement the 
reforms are not necessarily those who may have been able to participate in the application process. In 
accordance with federal SIG guidelines, the districts in which the 25 core sample schools are located 
officially led the SIG application process. 38 Overall, respondents in the 25 core sample schools described 
varying levels of opportunities for stakeholder involvement and input in the application process. 


For the three restart schools in the core sample, the respective CMOs assumed the role of the LEA in planning 
for its school's SIG application. 
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Box 6.4. School-Level Involvement in the SIG Application Process 

School classifications on the level of involvement of school staff in the SIG application process are 
described below (see Exhibit B.19 for more detail on the analytic procedures). For this analysis, 
"respondents" refers to the principal or at least the preponderance of evidence from the following 
respondent groups: principals, teachers, school improvement teams, and instructional coaches. 

Moderate school involvement ’ 

• Respondents reported that the principal had a significant role in the application process, 
typically leading the application process or working in collaboration with district staff to inform 
the subgrant application; OR 

• Respondents reported that the principal had a limited role, but other individuals at the school, 
such as members of the school improvement team, instructional coaches, or parent 
representatives, in addition to the principal, were involved in the SIG application process. 

Limited school involvement 

• Respondents reported that the principal (but no other school staff) provided feedback or input 
to inform the subgrant application, but had no role in planning. 

No school involvement 

• Respondents either reported that they had no involvement in the application process (the 
district applied for SIG and wrote the application), or that they had been unaware of the SIG 
application process. 

This analysis does not include a category for "significant involvement" as school-level respondents in none of the core sample 
schools described the principal and other school-level staff (i.e., school improvement team, instructional coaches ) as having a 
significant role in the application process. 

Nine core sample schools were classified as having moderate school involvement in the SIG 
application process, but the extent to which multiple school stakeholders were engaged in the process 
varied. One school, for example, reported widespread participation of school stakeholders. Respondents 
at this school indicated that the principal, who remained at the school in 2010-11, led the application 
process in cooperation with the superintendent. The principal brought in an external provider to help 
formulate the plans and also invited input from school staff. As noted by the union representative: 

[We] had a wonderful group of folks that really wanted to help our students here make this school 
a school that we could turn around. Because we had that open dialogue, and we had so many 
stakeholders, we were able to hear each other out... .We were able to come to a middle ground as 
we were writing the grants. ...Under our new leadership, we have engaged a lot of people. 

In other schools, the main contributors to the application were an assistant superintendent, an external 
provider, other central office staff, some lead teachers, and a school administrator who retired. At one 
school, the school improvement team, which included four instructional coaches, reported that they had 
some input into the grant application. As one coach described, "The district personnel came out one day 
and met with the leadership team, explained that we were [applying for] this grant, and listened to 
feedback about what we might need... .After that, the principal dealt with them." 

Ten core sample schools were classified as having limited involvement in the SIG application process, 
with participation typically limited to input from the current or previous principal. Among these 
schools are six in which the previous principal reportedly was involved in the SIG application process, 
even though he or she was later replaced as part of SIG. At one school, teachers reported that the 
previous principal had collaborated with district officials on the SIG application, even though teachers 
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voted against applying for SIG. In all teacher focus groups and interviews, respondents expressed 
frustration about the SIG application process. Teachers explained that they had collectively voted 
against participating in the grant competition because they felt that their school was already 
overextended with existing district demands. However, when teachers returned to campus in fall 2010, 
they were greeted by a new principal, who explained that the school would be participating in SIG. As 
one teacher explained, "It was like, 'Surprise! You got the grant!' and we were like, 'What? We thought 
we voted against that.'" Teachers at this school felt that their views were so much ignored that it was as 
if they never had a vote, but since the previous principal was reportedly involved in the process, the 
school was still classified as having limited involvement overall. 

Likewise, at another school, the former principal (during the 2009-10 school year) was reportedly 
involved in the application process. The new principal reported that she really had not known much 
about SIG until she started at the school in August 2010. Her initial reaction was that she welcomed the 
funds but that she wanted to make some revisions to the grant based on her own assessment of the 
school’s needs, but she was not able to do so. 

The remaining six core sample schools reported no involvement in the SIG application process. For 

example, respondents reported that the SIG application process at one school was top-down and 
district-led. One teacher explained, "I know the district did everything, but we were left in the dark. We 
weren't told why our school had to get rid of half our teachers, and we weren't told what we could do 
with the money." Two different teachers and the union representative at this school also reported that 
teacher input was not taken into consideration in drafting the SIG application. The majority of the 
teachers felt the SIG award was imposed on the staff, as teachers reported that the staff and community 
did not want to hire a new principal, for example. 

Timeliness of SIG Funding 

Whether or not SIG was received in a timely way could have affected schools' implementation of 
planned improvement actions and their use of SIG to support the change process. Early studies of SIG 
have noted that the timeliness of SIG funding was reported as a challenge for SIG-awarded districts and 
schools (Scott, McMurrer, McIntosh, & Dibner, 2012; U.S. GAO, 2011). As part of the application 
process, SIG funds passed through both SEAs and LEAs before arriving at SIG schools. SEAs submitted 
applications to ED; once ED approved a state's application for SIG, the state could then award grants to 
districts based on their applications for their eligible schools. The money then passed from the district to 
the school, although districts were able to retain up to 5 percent of the funding for grant-related 
expenditures. However, receipt of SIG at the school level could be delayed at any point as the funds 
funneled from the federal level to states to districts and finally to schools. State approval of district SIG 
applications took place in the summer of 2010 but, in some cases, extended into the fall of 2010. In one 
of the core sample districts, district officials required that school leaders request funding for specific 
expenditures rather than providing the full amount of the grant to the school all at once. 

Nearly half of the core sample schools (12 of 25) reported experiencing delays in the implementation 
of specific improvement actions because they did not receive SIG funding until after the start of the 
2010-11 school year. Delayed funding was most often reported as a constraint on schools' ability to hire 
new staff, finalize contracts with external support providers, and implement plans for increased learning 
time. Respondents at one school, for example, indicated that the district did not receive funds from the 
state until November 2010, and because no structure had been in place within the district to disburse 
the funds, the school was not able to access them until January 2011. As a result, the school deferred 
hiring an instructional coach, as planned in their SIG application, until the 2011-12 school year. 


97 


Case Studies of Schools Receiving School Improvement Grants 


An additional five core sample schools reported delays in receiving SIG funding, but respondents at 
these schools did not identify any associated implementation challenges. Respondents from two of 
these schools indicated that they worked with their districts to temporarily finance SIG-related activities 
through Title I or other funds. 

The remaining eight core sample schools received SIG funds on time at the start of the 2010-11 school 
year. 

Initial SIG Processes and Respondent Perceptions 

Initial SIG processes had the potential to provoke either positive or negative perceptions of SIG. These 
perceptions of SIG in the first year could have implications for how the grant is implemented in future 
years, although such perceptions could also change over time. 

In the four core sample schools where teachers conveyed negative perceptions of SIG, respondents 
reported that aspects of the application process had been mismanaged. For example, at one school, a 
district facilitator told teachers about SIG while they were attending a staff retreat intended to "build 
community." They learned that half of the staff would be replaced, an announcement that the principal 
described as "an atom bomb." At another school in which teachers had negative perceptions of SIG, the 
faculty had voted not to apply for SIG because they felt that it compromised the other reform efforts 
that were already under way. When teachers arrived at this school at the start of the 2010-11 academic 
year, they were surprised to find a new principal who was discussing SIG implementation. The principal 
explained that, "There was a lot of unhappiness. A year ago, they said 'we don't want it,' and yet the 
grant was written by the district and they said 'here you go anyway.'" 

In 10 of the other core sample schools, respondents described generally positive initial perceptions of 
SIG. Teachers, administrators, and parents explained that they welcomed the additional funds, staff, and 
anticipated technology purposes. One teacher recounted that a district administrator told them they 
would receive millions of dollars and that "You are going to decide at the building level what to do with it!" 

In the remaining 11 core sample schools, respondents described being unaware, ambivalent, or 
confused about SIG at the outset. In one such school, for example, teachers described learning about 
their school's SIG from a newspaper article that included erroneous information about the SIG models. 
As one teacher explained, "We had all sorts of questions and very few answers." 

In the four core sample schools where teachers described negative perceptions of SIG, SIG funds were 
delayed, and in three of these schools, the delay was reported to have caused problems. In one district, 
the district administrator explained: 

Folks really expected the money to be here in the fall, and of course that was through no fault of 
the district. ..the money [was] not released until October or November. So, people came to school 
thinking they were going to arrive to new pots of money and so there was some frustration 
about the fact that it took until the middle of the year to start seeing a trickle of things. And, 
everybody was trying to figure out how to roll it out. 

Funding delays also occurred in five core sample schools where respondents reported positive 
perceptions of SIG, so a delay did not necessarily curtail staff enthusiasm for SIG, even when the delay 
forced school administrators to postpone purchases or revise plans. 
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Initial Support for Implementation of SIG 

The core sample schools are situated within a variety of contexts and face a range of performance 
problems and resource constraints (see Chapter 3). To address these challenges and compensate for 
these constraints, states, districts, and external support providers may offer support to SIG schools. 
States and districts are responsible for helping schools understand how to launch and implement the 
turnaround process and how SIG can best be used to do so. The most recent authorization of ESEA 
required states to develop statewide systems of support for low-performing schools, moving the state 
toward a support and capacity-building role. Likewise, SIG encourages states and districts to move 
beyond the role of monitoring to one of support for the implementation of improvement initiatives. SIG 
guidance also prompts SIG schools to partner with external support providers— such as universities, 
nonprofit organizations, EMOs, or individual vendors— that work with schools to facilitate the 
improvement process. This section examines these potential sources of support for SIG implementation 
and the types of support that these sources reportedly provide. 

Sources of External Support 

Principals at most core sample schools (21 of 25) reported receiving at least some support from the 
state, district, or external provider (see Exhibit B.20 for more detail on the analytic procedures). 
Principals at the remaining four core sample schools, located in four districts and three states, described 
no support for SIG implementation. Principals at three schools reported receiving support from the 
state, whereas principals from 10 schools, located in seven districts, reported receiving support from 
their district, and 14 schools, located in eight districts, reported receiving support from external 
providers (Exhibit 6.3). Some districts (e.g., Districts 5, 9, and 12) appeared to rely on external providers, 
while other districts (e.g., District 7) appeared to mostly provide their own supports. 
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Exhibit 6.3. 

Support for SIG Implementation from States, Districts, 
and External Providers, by Core Sample School, 2010-11 






External Provider 



State Support for SIG 

District Support for 

Support for SIG 

District 

School Name 

Implementation 

SIG Implementation 

Implementation 



3 

10 

14 

District 1 

Blizzard Bay Elementary 



X 

District 2 

Aerovista High 

X 

X 


District 3 

Sherbrooke Elementary 


X 

X 

District 4 

Meribel High 





Baltimore Bridge Elementary 



X 

District 5 

Inner Brooks High 



X 

Sawbuck Elementary 
West Marble High 



X 


District 6 

Big Acorn High 
Peregrine Hill Elementary 

X 

X 

X 


Coral High 


X 


District 7 

Sterling Slope Elementary 


X 



Tyro Trail Elementary 


X 



Elmsville High 


X 


District 8 

Gillepsie High 

Raven Ridge Elementary 



X 


Island Bay Elementary 



X 

District 9 

McAlliston High 



X 


Tyron Elementary 



X 

District 10 

Melon Elementary 




District 11 

Haven Way Elementary 


X 


District 12 

Proctor Point High 

X 


X 

Rossignol Elementary 



X 

District 13 

Gale Secondary 


X 

X 

Paul Bunyan High 


X 

X 


Source: SST respondent interviews and focus groups, spring 2011. 

Note: Includes 25 core sample schools. All school names are pseudonyms. 


Types of Support From Districts and States 

States are allowed to reserve up to 5 percent of their SIG allocation to administer, monitor, and support 
SIG implementation. Although monitoring for compliance with legal requirements is an important aspect 
of the implementation of any federal program, recent emphasis has focused on providing the kinds of 
support that might increase the capacity of schools to launch and sustain improvement efforts. 

Principals at 20 of the 25 core sample schools (located in 11 of the 13 districts) reported that their 
districts monitored the implementation of their grant, and principals at 10 core sample schools 
(located in 7 districts) reported receiving district guidance on the technical aspects of implementation 

(Exhibit 6.4). 39 These supports included guidance on the appropriate use of funds and technical guidance 


Identified types of support are those reported by core sample school principals. Data from other school 
respondents, including teachers and instructional coaches, provided details about the types of support offered 
(see Exhibit B.21 for more detail on the analytic procedures). 
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on the requirements of the strategies in the SIG models, such as aspects of teacher evaluation or 
increased learning time. 


Exhibit 6.4. 

Types of District Support for SIG Implementation, by Core Sample School, 

2010-11 


District 

School Name 

Support for Implementation of 
Improvement Actions 

Compliance 
Monitoring and 
Guidance 

Data Analysis 
Support 

Teacher Professional 
Development 

Facilitation of a SIG 
School Network 

School Leadership 
Coaching or Support 

Guidance on 
Technical Aspects of 
SIG Requirements 

Monitoring SIG 
Implementation 



4 

8 

5 

6 

10 

20 

District 1 

Blizzard Bay Elementary 





X 

X 

District 2 

Aerovista High 


X 



X 

X 

District 3 

Sherbrooke Elementary 


X 



X 

X 

District 4 

Meribel High 






X 


Baltimore Bridge Elementary 






X 


Inner Brooks High 






X 

UISU ILL D 

Sawbuck Elementary 






X 


West Marble High 






X 

District 6 

Big Acorn High 






X 


Peregrine Hill Elementary 

X 

X 


X 


X 


Coral High 

X 

X 

X 

X 

X 

X 

District 7 

Sterling Slope Elementary 

X 

X 

X 

X 

X 

X 


Tyro Trail Elementary 

X 

X 

X 

X 

X 

X 


Elmsville High 



X 


X 

X 

District 8 

Gillepsie High 








Raven Ridge Elementary 






X 


Island Bay Elementary 





X 

X 

District 9 

McAlliston High 





X 

X 


Tyron Elementary 






X 

District 10 

Melon Elementary 






X 

District 11 

Haven Way Elementary 


X 






Proctor Point High 
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Rossignoi Elementary 





X 

X 

District 13 

Gale Secondary 


X 


X 




Paul Bunyan High 



X 

X 




Source: SST respondent interviews and focus groups, spring 2011. 

Notes: Includes 25 core sample schools. All school names are pseudonyms. 


Principals at 10 core sample schools (located in 7 of the 13 districts) reported receiving support for 
improvement and improvement actions that went beyond compliance monitoring and guidance. For 

example, eight schools reported receiving support for teacher professional development. Six schools in 
three districts reported coaching or support focused on school leadership. Five schools located in three 
districts reported that they participated in a district-facilitated network of SIG schools in which leaders 
or staff convened regularly to discuss SIG implementation, receive information, and share practices and 
challenges. 
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Core sample schools in the same district often described receiving different levels of district support (5 
of 7 districts with multiple core sample schools). In some cases, there was an explanation for the 
differences. For example, in one district where schools are implementing different SIG models, the 
restart school may not have received support from the district because the CMO or EMO may be playing 
that role. In two other districts, the elementary school reportedly received more support than the high 
school, although in general it was not always the case that our core sample elementary schools received 
more support than the high schools. 

District administrators reported providing support for improvement more often than school principals 
reported receiving such support (see Exhibit 6.5). For example, district administrators from 10 districts 
serving 16 core sample schools reported providing professional development to teachers, but only eight 
core sample school principals in two districts reported receiving teacher professional development. 
District administrators also more often reported monitoring activities in SIG schools than was reported 
by the core sample school respondents. For example, 12 districts serving 24 core sample schools 
reported that they monitored SIG implementation for compliance with federal and state guidelines. 
However, principals at only 20 core sample schools reported receiving this type of monitoring. (See 
Exhibit B.21 for more detail on the analytic procedures.) 


Exhibit 6.5. 

Types of District Support for SIG Implementation 

as Reported by District Administrators and Principals in Core Sample Schools, 2010-11 


SUPPORT FOR SIG IMPLEMENTATION 


Data Analysis Support 
Teacher Professional Development 
Facilitation of a SIG School Network 
School Leadership Coaching or Support 

COMPLIANCE MONITORING AND GUIDANCE 

Guidance on Technical Aspects of SIG Requirements 

Monitoring SIG Implementation 



■ Number of Core Sample Schools Receiving This Type of District Support as Reported by District Administrators 
Number of Core Sample Schools Receiving This Type of District Support as Reported by the Principal 


Source: SST respondent interviews and focus groups, spring 2011. 
Notes: Includes 25 core sample schools. 


State support to districts (and directly to three of the core sample schools) reportedly focused on 
compliance monitoring. State officials in the six states, where core sample schools were located, 
reported during our interviews with them that they primarily worked with districts and focused on 
compliance-oriented activities. For example, officials in all six states reported that they responded to 
questions about SIG compliance. District administrators in 9 of the 13 core sample districts similarly 
reported that their interactions with state officials were limited to compliance monitoring rather than 
support for SIG implementation and improvement actions. 
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Reports of State and District Capacity 

Although states and districts are responsible for the monitoring and support of SIG schools (and may 
reserve some of their total allotment of SIG funds to do so), state and district capacity may shape the 
actual amount and quality of support that schools receive (Le Floch, Boyle, & Therriault, 2008). This 
includes the number of staff available, the knowledge and skills of the staff, and the procedures in place 
to facilitate both standard compliance as well as efficient enactment of changes. 

The state officials and district administrators we interviewed reported challenges to fulfilling their roles 
in administering the SIG program and providing support to SIG schools. They reported a lack of capacity 
for supporting SIG implementation due to declining budgets or recent budget cuts, along with limited 
numbers of staff. District administrators also reported struggling to balance the tension between SIG 
schools and other schools in their districts. 

ESEA requires states to establish a statewide system of support for low-performing schools, which states 
used as a platform for providing support to SIG schools (see Hurlburt et al., 2011). In this study's sample 
of six states, officials from three reported that their agencies reorganized or planned to reorganize to 
better support districts and SIG schools. The other three states planned to rely on their existing system 
of support. 


Box 6.5. Perceptions of State and District Capacity to Support SIG Schools 

Classifications of state and district capacity are described below (see Exhibit B.22 for more detail on the 
analytic procedures). For this analysis, "respondents" refer to state and district administrators. Where 
discrepancies were noted between the state and district respondents, analysts deferred to self- 
reports— that is, state respondents for state capacity and district respondents for district capacity. 

Capacity to support SIG implementation 

• Respondents indicated that the state/district had the capacity to support SIG implementation. 

Limited or no capacity to support SIG implementation 

• Respondents indicated that the state/district had limited or no capacity to support SIG 
implementation. 

Five of the six state officials interviewed described having limited staff capacity to support SIG 
implementation, an issue that was reportedly exacerbated by the decline in fiscal resources. As one 
official summarized: 

This is one of the biggest school reform efforts I've been involved with. ...SIG is huge, and the fact 
that we don't have any more staff than we had 10 years ago is just amazing... .It is difficult 
because we keep getting more grant programs or more monitoring data reguirements from the 
federal government, and we get no more staff. 

A second state official explained that lack of staff limited the state's capacity to focus on improvement 
(versus monitoring and compliance-related) activities: "Staff who were supposed to work with districts 
and schools in the field [are] required to work in the office to do paperwork and focus on grant 
renewal." A third state official explained: "We don’t have a lot of resources. ..to implement and more 
importantly to give effective guidance [to districts and schools]." In addition to shortchanging support 
activities, state officials reported that capacity limitations hindered the identification of eligible schools, 
developing a district application process, assessing district capacity, and conducting the SIG review 
process and awarding funds. 
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Reports from state officials and district administrators suggest that district capacity to support SIG 
implementation (e.g., monitoring, guidance, or improvement strategies) also was strained. State 
officials in each of the six states indicated that district capacity to implement SIG was limited. One state 
official described a process through which the state accepted the assurance of the district "that [it] 
could and would implement [SIG] fully," and during the state reviews of districts, they found that "many 
aspects hadn't been correctly implemented or fully implemented." Administrators from six districts in 
four states serving nine core sample schools reported having diminished capacity due to turnover in 
district staff and budget cuts. One district superintendent stated, "We're operating at bare bones." 

District staff sought to improve support to SIG schools by reorganizing the district office, developing a 
special district unit focused on SIG schools, and enhancing existing school support strategies. 

Administrators from 6 of the districts serving 14 core sample schools recently reorganized or planned to 
reorganize district departments to improve service to SIG schools. For example, one large urban district 
created a special office dedicated to the support and monitoring of SIG-funded schools. In another 
district, a budget administrator was hired, and a school improvement facilitator was reassigned from 
another district position with the primary responsibility of supporting SIG schools, although the nature 
of the support was still being defined. Five districts reported creating an office or even a special 
subdistrict to provide focused attention, support, and monitoring to SIG-funded schools. 

By reorganizing, district administrators hoped to improve access to district resources and staff. 

However, respondents from 4 of the 14 core sample schools located in districts that reorganized or 
planned to reorganize to support SIG schools reported receiving little or no support for school 
improvement from their districts. Respondents from 4 core sample schools in districts that reorganized 
also reported that the district was a barrier to SIG implementation. For example, the principal and 
teachers at one school, located in a district that reorganized, explained that the district impeded their 
improvement efforts because district administrators denied funding for improvement strategies, even 
those that were research-based, or the district took a long time to approve SIG-related expenditures, 
making it difficult to implement improvement strategies. 

Additionally, perhaps reflecting limited state and district capacity, 14 of the 25 core sample schools 
reported receiving support from external partners, and 8 of the 13 districts relied on external partners 
to provide some support in one or more of their SIG schools (Exhibit 6.3). 

The type and quality of support that schools reported receiving from external providers differed 
across schools. In the three restart schools, the CMO/EMO was the external provider that managed the 
school and supported the implementation of SIG. However, respondents in one restart school reported 
almost no contact with the CMO that had assumed management of the school. The remaining 11 
turnaround and transformation schools reported receiving leadership coaching, professional 
development, or instructional support (in the form of instructional coaches) from an external provider. 
School respondents reported that the quality of the support varied. Teachers and administrators in six 
schools reported that the external provider was very supportive of the school's work. In contrast, 
respondents from four schools did not find the support offered by the external provider to be helpful. 

Chapter Summary 

SIG intended to induce persistently low-performing schools to take bold action that would disrupt 
business-as-usual. As we have seen in Chapters 5 and 6, the 25 core sample SIG schools, consistent with 
the requirements of the four SIG intervention models, replaced principals, hired new teachers, increased 
learning time, or implemented other actions intended ultimately to improve student outcomes. In seven 
schools, these visible changes, taken together, were experienced as a disruption from the past. 
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At the same time, one of the criticisms of the four SIG intervention models is that they are too restrictive 
and prescriptive (see Klein, 2012), and not sufficiently sensitive to the contextual conditions of each 
school. Data from our study, however, suggest that many of our districts were able to and did consider 
local constraints (such as the labor supply, union presence, or current leadership) when selecting SIG 
models, and many of our core sample schools did find ways to use SIG to enhance reform activities that 
had been planned at the local level. Indeed, SIG was reported to fit into a change process that had been 
planned or launched prior to SIG in 19 of our 25 core sample schools. 

The initial implementation process for SIG was not without challenges. Respondents in most core 
sample schools reported that their involvement in the SIG application process was either limited (10 
schools) or nonexistent (6 schools). Likewise, there were reportedly glitches in disbursing SIG funds. 
Respondents at 17 of the 25 schools described delays in funding, although just 12 of these schools 
reported that the delays in funding also resulted in delays to implementation of SIG activities. 

To support the challenging implementation process, SIG specified that their grantee schools were to 
receive supports for their improvement efforts. Respondents from 21 core sample schools reported 
receiving assistance from the state, district, or an external support provider. For most schools, this 
assistance consisted of monitoring rather than direct assistance with implementing school improvement 
activities. 
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Chapter 7: Leading Indicators of Change 


The purpose of this study is to explore the change process in our core sample of persistently low- 
performing SIG schools. At the end of the 2010-11 school year, respondents from these schools 
described whether they had witnessed improvements over the course of the year, although it is 
important to remember that such changes are not necessarily attributable to SIG. 

This chapter describes respondents' assessments of their schools' progress— or lack thereof— after the 
first year of SIG. Although this analysis does not include direct measures of student achievement, 
respondents' comments do address aspects of school climate and functioning that in previous studies have 
been associated with higher-than-expected student achievement (see Chapter 1). Because of these 
associations, researchers have considered such organizational characteristics to be indicators of a school's 
capacity to produce high levels of desired student outcomes (Beaver & Weinbaum, 2012; Corcoran & 
Goertz, 1995; Newmann, King, & Youngs, 2000). In the context of a school improvement initiative, 
increases in these indicators may be viewed as signs of increased organizational capacity and thus as 
potential precursors to— or leading indicators of— subsequent improvements in student achievement. This 
chapter will explore perceived improvement in these hypothesized leading indicators during the first year 
of SIG, as well as perceived levels of organizational capacity at the end of the first year. 

Specifically, we examine two topics: (1) the extent to which respondents from core sample schools 
perceived that progress after the first year of SIG implementation; and (2) how many, and which, 
schools demonstrated capacity for high performance (as defined by evidence of the leading indicators 
hypothesized in SST's conceptual framework [see Exhibit 1.2]). We also explore the relationship 
between school capacity and contextual variables described in Chapter 3, including external context, 
level of SIG funding, and SIG model. Finally, we revisit the topics of strategic leadership (Chapter 4), 
teacher replacement (Chapter 5), and schools that experienced a disruption from the past (Chapter 6), 
and consider how they may be related to reports of improvement and school capacity. 


Box 7.1. Key Chapter 7 Findings 

• Respondents in all but one of the 25 core sample schools reported some improvement in 2010- 
11. The most frequent reports of improvement related to school climate, staff collaboration, 
and instructional practices. 

• The core sample schools in which respondents described improvements in the greatest number 
of areas also had higher levels of principal strategic leadership (Chapter 4) and were more likely 
to have experienced a disruption from the past (Chapter 6). 

• The core sample schools with the lowest levels of organizational capacity (seven schools) were 
those in which teachers reported fewer resources, in which SIG awards were larger relative to 
the prior year's per-pupil expenditures, and in which SIG was perceived as an impetus for 
change. 


Respondents' Perceptions of Progress 

SIG intends to be a substantial intervention for persistently low-performing schools, catalyzing dramatic 
action and yielding immediately perceptible improvements. Although better student outcomes may not 
be expected after only one year of implementation, our site visit data contain reports of improvement in 
elements of organizational functioning and educational practice during the 2010-11 school year. 
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This section summarizes respondents' perceptions of improvement in Year 1 of SIG for hypothesized 
leading indicators in SST's conceptual model (e.g., principal leadership, safe and orderly climate, data 
use), and explores how perceptions of improvement are associated with other school characteristics. 
We include data from interviews and focus groups with district administrators, principals, teachers, 
instructional coaches, parents, and students. We did not examine whether schools had actually 
improved student test results, graduation rates, or other measures of student outcomes. Rather, we 
focused on whether respondents reported improvement on indicators of school organization and 
practice that the literature frequently associates with improved student outcomes (Bryk, Sebring, 
Allensworth, Luppescu, & Easton, 2010). Respondents' perceptions of these indicators could potentially 
reflect objective conditions in the school, but even if they do not align with more objective measures, 
these perceptions could nonetheless influence how respondents subsequently relate to and participate 
in the improvement process. For example, teachers and administrators who perceive initial 
improvement may become more committed to the reforms, may increase their own perceptions of 
efficacy, and may increase their motivation to pursue the improvement process. Such patterns are 
consistent with theory on employee motivation and organizational change (Lawler, 1994; Mohrman & 
Lawler, 1996). 


Box 7.2. Perceived Improvement in Core Sample Schools 

School classifications on perceived improvement are described below (see Exhibit B.23 for more detail 
on the analytic procedures). Schools were classified as having made improvement on a leading indicator 
if at least one of the following respondents— teachers or the principal— and at least one respondent in 
one other respondent group (i.e., district administrator, instructional coach, parents, students) 
described improvement for the indicator, and no respondents disagreed or made statements to the 
contrary. For teacher collaboration, the threshold for perceived improvement was at least two teachers. 
Reports of improvement in many areas 

• Perceived to have made improvement for at least six of the eight indicators; AND 

• Respondents described improvements in strong, illustrative language, such as a "light going on" 
or a "rebirth." 

Reports of improvement in some areas 

• Perceived to have made improvement for three to five of the eight indicators. 

Reports of improvement in few or no areas 

• Perceived to have made improvement for less than three of the eight indicators. 


Degrees of Perceived Improvement 

Respondents at 24 of the 25 core sample schools reported improvement in at least one area, with 6 
schools reporting improvement in many areas, 8 schools in some areas, and 10 schools in few areas 

(see Box 7.2). 

In each of the six schools that reported improvement in many areas (i.e., 6 to 7 areas), respondents used 
strong descriptive language to characterize the improvements. For example, at one school, individuals 
from five respondent groups (district administrator, principal, teachers, instructional coach, and 
students) reported improved instruction, staff expectations for students, student engagement, staff 
collaboration, school climate, leadership, and data use. In describing this high school, the district 
superintendent stated, "There is positive [change] in just one year. We have a long way to go, but the 
point is there has been a U-turn in the right direction." This perspective was shared by respondents 
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(administrators, teachers, coaches) within the school as well. The principal explained, "I keep saying this 
to folks, this [district reform] piece and the School Improvement Grant has absolutely been the best 
thing that has happened in the life of this school for at least the last 35 to 40 years." 

In the eight schools that reported improvement in some areas, respondents indicated progress on three 
to five of the leading indicators. Respondents also pointed out ongoing challenges or internal conflicts. 
For example, at one school, individuals from five respondent groups (principal, teachers, instructional 
coaches, parents, and students) reported improved safety and orderliness at the school, as well as 
improved teacher collaboration and data use. Despite perceived progress on these indicators, there 
were conflicting views among school staff, as teachers described a lack of direction and tension among 
teachers in different departments. In addition, teachers from this school, who participated in focus 
groups, reported a lack of visible improvement. 

In the 11 schools that reported improvement in few or no areas, respondents indicated progress on no 
more than two of the leading indicators. In one of these schools, teachers said that the school had 
regressed during the first year of SIG implementation. One teacher reported: 

What I perceived to be the culture of the school last year was this great place where we had 
members of the community that were really attached to [the school]... unfortunately, I would 
have to say that has been pretty much lost this year. 

Domains of Perceived Improvement 

The most common areas of perceived improvement were with regard to a safe and orderly climate (14 
schools), teacher collaboration (13 schools), instructional practices (12 schools), and quality of 
leadership (12 schools) (see Exhibit 7.1). For example, teachers, parents, and the principal at one school 
all commented on the improvement in student behavior and safety. A parent mentioned that "the 
security is great," and that although it used to be a problem, the issue "has been addressed." One 
teacher explained, "Most people who visit the school find it to be a functioning, quiet, organized place," 
while another mentioned that "the number of suspensions is definitely down." 

Teacher respondents in 13 schools reported improvements in teacher collaboration. As a teacher at one 
school explained, "I have amazing, supportive professional relationships with other teachers. It is not 
just with a small group of teachers, like only females or only math teachers, it is across the board. We 
have been given a lot of opportunities during professional development and common planning time to 
build relationships, so when things get tough and stressful, we can go to each other." In contrast, 
teacher collaboration reportedly decreased at another school in 2010-11: "Another thing that is very 
frustrating is the lack of teaming. That goes into the scheduling problem. There are no more core 
planning periods," reported one teacher. 

Respondents in 12 schools described improvement in instructional practices. Seven of these schools had 
replaced teachers in 2010-11. For example, district officials, the principal, teachers, parents, and 
students at one school described improved instruction. One district official observed: 

Two years ago I was still having to say, "Where is the standard? Let me see the lesson plan." 

[When visiting the school] this year, every classroom matches the lesson when they are teaching, 
and they are doing the benchmark assessments. They caught up in a year what probably would 
have taken three years at the pace we were going. 

Students at this school also described improvements in the classroom, particularly mathematics, in part 
because the entire mathematics department was new in 2010-11: 

[Mr. X] takes the time. He doesn't leave a student behind. If one student doesn't get it he 


108 


Case Studies of Schools Receiving School Improvement Grants 


doesn't move on until everyone understands... .He explains it to you and takes time to explain the 
steps. 

He shows us different methods and gives us options to use to solve the problems. 

[Last year,] [the teachers] would rush through it. If one person understood, they would rush 
through it. No wonder I failed geometry twice. 


Exhibit 7.1. 

Number of Core Sample Schools with Perceived Improvement on 
Specified Leading Indicators, 2010-11 


Safe and Orderly Climate 
Teacher Collaboration 
Instructional Practices 
Quality of Leadership 
Use of Data for Instructional Decisions 
Material Resources 
Student Engagement 
Staff Expectations for Students 



0 5 10 15 20 25 

Number of Core Sample Schools Reporting Improvement 


Source: SST respondent interviews and focus groups, spring 2011. 
Notes: Includes 25 core sample schools. 


Among the 12 core sample schools that reported improvement in instructional practices, 9 also 
reported improvement in teacher collaboration. For example, one restart school with a completely new 
staff instituted new practices that shaped instruction and collaboration. Each school day began with a 
"zero period" that was designated for staff collaboration. Teachers used this period to discuss lesson 
planning, curriculum, data analyses, and instructional practices. 

Five core sample schools reported improvements in staff expectations for students. For example, a 
teacher at one school described a change in her students in response to her higher expectations: "When 
I got here, the kids just didn't care about school. But when I raised the standards and the bar, they were 
able to do it." Another teacher at that school agreed: "Now that we have new staff and new 
expectations. ..they are actually more receptive to it than you would guess from a kid that went from not 
doing much to now all of a sudden you have to do your homework, you have to study for tests. They like 
it, they seem to like it." In another school, the principal explained: "One kid commented that last year's 
staff would make a silent agreement that as long as you didn't make problems in class, they would let 
you pass. This year's staff will not allow this to happen." Although respondents at most of the 25 core 
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sample schools did not report improvements in this area, it is important to note that unlike the other 
areas, they were not explicitly asked to report on this specific area of improvement. 

Aligning Improvement Actions With Perceived Improvements 

Given that a focus of this chapter is on describing perceived improvements, it is of interest to try to learn 
more about the characteristics of our core sample schools that perceived improvements. We begin by 
examining how many of our schools that perceived improvements in specific areas also reported 
implementing corresponding improvement actions in these areas. 

Respondents in core sample schools that described improvements in school climate, teacher 
collaboration, instructional practices, and data use often reported that they also implemented 
improvement actions in these areas (see Exhibit 7.2 and Chapter 5). For example, respondents from all 
12 schools that reported improvement in teacher collaboration and instructional practices also reported 
implementing corresponding improvement actions, such as hiring an instructional coach or increasing 
professional development. Respondents from 9 of the 14 schools that reported improvements in school 
climate also described improvement actions focused on student behavior. For example, one school 
reportedly implemented policies to improve attendance, began using new uniforms, and adopted new 
rules and procedures concerning student behavior. As one member of the school improvement team at 
this school noted, "Everything is being done. We [implemented] new routines and procedures. You will 
tuck in your shirt; you will not spit in the stairwell. It was culture shock. They knew that every face was 
new, that walls had bright paint, and that we cleaned up as best we could." 

Respondents in some schools that described implementing improvement actions in these four areas did 
not also report corresponding improvements. Nevertheless, our results generally indicate that where 
schools reported improvement, they also reported implementing corresponding improvement efforts. 
We are unable to determine whether this is a causal connection, however (i.e., that implementing these 
improvement actions led to improvements). 
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Exhibit 7.2. 

Reports of Perceived Improvement and Corresponding Improvement Actions in 
Core Sample Schools for Select Leading Indicators, 2010-11 


Safe and Orderly Climate 


Teacher Collaboration 


Instructional Practices 


Use of Data for Instructional Decisions 


14 

9 



20 


25 


■ Number of Core Sample Schools Reporting Improvement 
Number of Core Sample Schools Reporting a Corresponding Improvement Action 


Source: SST respondent interview and focus groups, spring 2011. 

Notes: Includes 25 core sample schools. Each light blue bar contains a subset of the schools in the corresponding dark blue bar. 
For example, 14 schools perceived having a safer and more orderly climate, from which 9 also reported implementing 
improvement actions to promote a safe and orderly climate. Quality of leadership, material resources, student engagement, 
and staff expectations for students were excluded, as there were no improvement actions clearly associated with these leading 
indicators. 


Association Between Perceived Improvement and School Characteristics 

Respondents in core sample schools with higher levels of strategic leadership (see Chapter 4) or that 
had experienced a disruption from the past (see Chapter 6) reported improvement in more areas than 
did schools without these characteristics. However, the degree of perceived improvement was 
unrelated to a host of other characteristics described in earlier chapters of this report, including: which 
SIG model the school implemented, whether the school was a high school or elementary school, 
whether the school was in a benign, depressed, or traumatic context, whether the school experienced a 
delay in receiving SIG, whether the school perceived funding levels to be a barrier to improvement, the 
school's level of stakeholder involvement in their SIG application process, teachers' reported level of 
trust in the principal, the total number of improvement actions the school reported implementing, and 
the size of the school's SIG award relative to per-pupil funding in the prior year. 
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Exhibit 7.3. 

Number of Leading Indicators with Perceived Improvement, 
by Core Sample School and Principal's Strategic Leadership, 2010-11 


Inner Brooks High 



West Marble High 
Baltimore Bridge Elementary 




Haven Way Elementary 

in many areas 

Meribel High 


Sterling Slope Elementary 


Blizzard Bay Elementary 


Peregrine Hill Elementary 


Sawbuck Elementary 


Sherbrooke Elementary 


Big Acorn High 


Coral High 


McAlliston High 


Rossignol Elementary 

J 

Aerovista High 


Gale Secondary 


Gillepsie High 


Island Bay Elementary 


Melon Elementary 


Proctor Point High 

Reports of improvement in 
few or no areas 

Elmsville High 

Paul Bunyan High 


Raven Ridge Elementary 


Tyron Elementary 


Tyro Trail Elementary 

High on continu 
■ Mid-low on cont 

1™ J 

>12345678 

Number of Leading Indicators with Perceived Improvement 

jm of strategic leadership Mid-high on continuum of strategic leadership 

inuum of strategic leadership | Low on continuum of strategic leadership 


Source: SST respondent interviews and focus groups, spring 2011. 

Notes: Includes 25 core case study schools. All school names are pseudonyms. Strategic leadership measures are drawn from 
analyses presented in Chapter 4. Tyro Trail Elementary reported improvement in no areas and was categorized as having a low 
level of strategic leadership. 


All 6 core sample schools with reports of improvement in many areas had principals rated as having 
moderately-high or high levels of strategic leadership, while 10 of the 11 schools with reports of 
improvement in few or no areas had principals with moderately-low or low levels of strategic leadership 
(see Exhibit 7.3). 

Among the 6 core sample schools that reported improvement in many areas, 5 were classified as having 
experienced a disruption from the past in 2010-11 or 2009-10, while none of the 11 schools that reported 
improvement in few or no areas were classified as having experienced such a disruption (see Exhibit 7.4). 

Hence, although two important aspects of the SIG program— new leadership and an impetus for swift, 
dramatic change— were related to subsequent perceptions of improvement in the first year of 
implementation, many other factors that were hypothesized to be relevant to SIG (such as which SIG 
model was implemented) did not appear to be related to the levels of perceived improvement in our 
core sample schools. 
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Exhibit 7.4. 

Level of Perceived Improvement in Core Sample Schools, 
by Reports of a Disruption From the Past 



Schools With Reports of 
Improvement in Many 
Areas 

Schools With Reports of 
Improvement in Some 
Areas 

Schools With Reports of 
Improvement in Few or 
No Areas 

Total 

Experienced 
Disruption 
From the 
Past 

5 schools 

Baltimore Bridge Elementary 
Inner Brooks High 
Meribel High 

Sterling Slope Elementary 
West Marble High 

2 schools 
Big Acorn High 
Sawbuck Elementary 


7 

schools 

Did Not 
Experience 
Disruption 
From the 
Past in 

2009- 10 or 

2010- 11 

1 school 

Haven Way Elementary 

6 schools 

Blizzard Bay Elementary 
Coral High 
McAlliston High 
Peregrine Hill Elementary 
Rossignol Elementary 
Sherbrooke Elementary 

11 schools 
Aerovista High 
Elmsville Elementary 
Gale Secondary 
Gillepsie High 
Island Bay Elementary 
Melon Elementary 
Paul Bunyan High 
Proctor Point High 
Raven Ridge Elementary 
Tyron Elementary 
Tyro Trail Elementary 

18 

schools 

Total 

6 schools 

8 schools 

11 schools 



Source: SST respondent interviews and focus groups, spring 2011. 

Notes: Includes 25 core sample schools. All school names are pseudonyms. Measures of disruption from the past are drawn 
from analyses presented in Chapter 6. 


Stories of Improvement 

Although there are features common to our core sample schools that reported more improvement— 
and to our core sample schools reporting little or no improvement— each school represents a unique 
story of improvement. 

For example, one of our core sample schools is a case of alignment between the SIG turnaround model 
and the school's performance problem. It is a small school in a rural area, an hour's drive away from 
shopping, entertainment, medical services, a library, or any other features common to most cities or 
towns. According to the principal, teachers, and district administrators, this school had for years been 
the "forgotten stepchild" of the district and the "dumping ground" for the district's lowest-performing 
teachers. Indeed, respondents in interviews and focus groups frequently used these two phrases to 
describe the school. In prior years, instruction was reported to be nearly nonexistent, as teachers played 
videos in class, did not assign homework, and did not administer classroom tests. However, this school 
had some favorable conditions: although the community was impoverished, it was reportedly "as safe as 
any suburban community," and the students were reported to be well behaved, albeit apathetic about 
their education. The turnaround model gave administrators the authority to replace half of the teachers, 
including the entire mathematics department, resulting in a reportedly dramatic change in instructional 
quality, as well as improvements in several other areas. In addition, teachers at this school had positive 
perceptions of SIG and reported high levels of data use and collaboration. In summary, this school 
appeared to leverage SIG effectively because the school's most salient challenge— underqualified 
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teachers and poor instruction— fit neatly with the key requirement of the SIG turnaround model, which 
is to replace 50 percent of the teachers. 

In contrast, another one of our core sample schools reported facing numerous challenges, with few 
assets and little improvement as of 2010-11. An urban elementary school situated in a challenging 
environment, it is located in a neglected, high-poverty, and reportedly "unsafe" part of the city, 
described by a teacher as "one of the most dilapidated and worst areas in [the city]." Respondents 
explained that their school has historically been a "dumping ground" for low-performing teachers and 
students who were not successful in other settings, such as charter and magnet schools. A teacher 
explained that kids at this school "don't know their birthdays. ..they just know survival skills." According 
to teachers, instructional coaches, and parents, the school principal (who arrived in 2009-10) was an 
"authoritarian" leader and at times "intimidating," with some teachers indicating that the principal 
created an uncomfortable and distrustful culture. Moreover, as reported by the principal, improvement 
actions associated with SIG were in their infancy, including an intervention designed to improve student 
behavior and school culture that was not to be implemented until the 2011-12 school year. On the 
positive side, the school is situated in a district that offered multiple supports to SIG schools and 
communicated with schools frequently about their needs. Nevertheless, the leadership and largely 
inexperienced staff did not operate cohesively during the first year of SIG, according to respondents. 
Several factors— a difficult external environment, a chaotic internal environment characterized by a lack 
of trust, authoritarian leadership, and inexperienced teachers— appeared to hinder progress. 

School Capacity to Improve Student Learning 

Reports of improvement - even widespread improvement - do not necessarily mean that a school has 
built the capacity necessary to foster and sustain high levels of student achievement. O'Day, Goertz, and 
Floden assert that "within the context of systemic reform, capacity is the ability of the education system 
to help all students meet more challenging standards" (1995). Hatch (2009) takes a similar approach to 
what he calls a school's "collective competency" to improve student outcomes. As depicted in SST's 
conceptual framework (see Chapter 1), we identified a set of eight intermediate outcomes that are 
thought to be associated with higher levels of student achievement, and which together may reflect a 
school's organizational capacity. Although the many studies from which these variables derive are 
primarily correlational in nature, the consistency of findings across these studies suggests that, taken 
together, these variables may be indicative of an overall level of organizational capacity. 

We thus now explore the status of each core sample school on a set of eight indicators of capacity: 
leadership, coherence, clear and shared goals, teacher collaboration, teacher-teacher trust, safe and 
orderly climate, use of data to inform instruction, and the extent to which respondents described an 
internal or external responsibility for performance problems. The definition and research base for each 
indicator is presented in Chapter 1, and Exhibit 7.5 provides an overview of how we measured each 
indicator. Two of these indicators of capacity are described earlier in this report: internal and external 
responsibility for performance problems (Chapter 3) and leadership (Chapter 4). The remaining 
measures, summarized in Exhibit 7.5, are based on survey data and/or qualitative data, and the ones 
that include qualitative data (i.e., teacher collaboration, safe and orderly climate, and use of data for 
instructional decisions) are described in greater detail in Exhibits B-24, B-25, and B-26. 

The analyses that follow first consider the presence and levels of these individual elements of 
organizational capacity in each of the core sample schools. The analyses then combine the elements into 
a composite measure of school capacity and explore relationships between this composite measure and 
other characteristics of the schools and their environments. This approach is similar to that of Beaver 


114 


Case Studies of Schools Receiving School Improvement Grants 


and Weinbaum (2012) who recently identified four literature-based components that together reflect 
school capacity for improvement: human capital, social capital, program coherence, and resources. 
Although the specific terminology we use in this report differ somewhat from that of Beaver and 
Weinbaum, there is considerable overlap, and the variables we use can fit neatly into their proposed 
framework. 


Box 7.3. Organizational Capacity in Core Sample Schools 

For each indicator described in Exhibit 7.5, analysts assigned numeric values to the classifications (0 for 
the lowest category, 1 for the middle category, and 2 for the highest category), which were summed to 
create an aggregate index of school capacity. School classifications on the overall level of organizational 
capacity are described below (see Exhibit B.27 for more detail on the analytic procedures). 

Higher capacity 

• Received a summative rating of at least 10 out of 16 on the school capacity index. 

Moderate capacity 

• Received a summative rating of 8 or 9 out of 16 on the school capacity index. 

Lower capacity 

• Received a summative rating of less than 8 out of 16 on the school capacity index. 

Schools with higher ratings on these eight measures can be described as having greater overall school 
capacity relative to the other schools in our core sample. For example, we hypothesize that one of our 
core sample schools with a principal rated highly on all three measures of leadership, with high levels of 
teacher trust, with better than average programmatic coherence, and stronger data use practices will be 
in a better position to improve student outcomes than one of our core sample schools with low ratings 
on all of these measures. 

There are a few important caveats to consider when interpreting these data. Our study design did not 
permit development of a pre-SIG measure of capacity, and so our results should be interpreted as 
providing a baseline of organizational capacity for future improvement and performance. Moreover, our 
analysis omits several important measures of organizational capacity for which the study team did not 
have adequate data, such as teachers' content knowledge and quality of instruction. 
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Exhibit 7.5. Indicators of School Capacity 


Leading Indicator 

Data Sources 

Classifications 

Leadership (see 
Chapter 4 and 
Exhibits B.5-B.7 

Qualitative data 

• Interviews with the principal, teachers, and 
instructional coaches 

• Focus groups with teachers and school improvement 
teams 

Survey data 

• Principal trust and instructional leadership survey scales 
*Two core sample schools had insufficient data on 
transformational and instructional leadership, so their 
classification is based solely on strategic leadership. Using 
this approach, one was rated as moderate level and the 
other as low level. 

High level (2 schools) 

• The principal was classified as "high on continuum" across all three leadership dimensions 
(transformational, instructional, and strategic). 

Moderate level (16 schools) 

• The principal was not classified as "high on continuum" or "low on continuum" in more 
than two of the three leadership dimensions (transformational, instructional, and 
strategic). 

Low level (3 schools) 

• The principal was classified as "low on continuum" across all three leadership dimensions 
(transformational, instructional, and strategic). 

Insufficient data (4 schools) 

Coherence (see 
Exhibit B.27) 

Survey data 

• Three teacher survey items* for which respondents 
agreed or disagreed with the following statements: 

1. "Once we start a new program, we follow up to make 

sure that it's working." 

2. “1 worry that we are adopting too many different 

programs and practices in this school."** 

3. “This school generally chooses only those school 

improvement activities that fit with our improvement 
goals and strategies." 

*Although these items all measure aspects of coherence, 
they did not constitute a reliable scale. Thus, analysts 
developed an index based on whether schools were 
consistently above or below the mean on each. 

**Because this item was negatively worded, it was 
reverse-coded for purposes of analysis. 

High (3 schools) 

• Received a summative rating of at least 7 out of 9 on the coherence index (For each item, 
schools were assigned numeric values based on the school mean relative to the overall 
mean (l=at least 0.5 standard deviations below the overall mean; 2=within 0.5 standard 
deviations of the overall mean; 3=at least 0.5 standard deviations above the overall 
mean), which were summed to create a coherence index.). 

Medium (12 schools) 

• Received a summative rating of 6 out of 9 on the coherence index. 

Low (6 schools) 

• Received a summative rating of less than 6 out of 9 on the coherence index. 

Insufficient data (4 schools) 
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Exhibit 7.5. Indicators of School Capacity (continued from previous page) 


Leading Indicator 

Data Sources 

Classifications 

Clear and Shared 
Goals (see Exhibit 
B.27) 

Survey data 

• Shared goals survey scale 

High level (2 schools) 

• Schools with a mean response more than 0.5 standard deviations (0.30) above the overall 
mean (3.18). 

Moderate level (16 schools) 

• Schools with a mean response between 0.5 standard deviations (0.30) below and 0.5 standard 
deviations above the overall mean (3.18). 

Low level (3 schools) 

• Schools with a mean response 0.5 standard deviations (0.30) below the overall mean (3.18). 
Insufficient data (4 schools) 

Teacher 

Collaboration (see 
Exhibit B.24) 

Qualitative data 

• Interviews with principals, teachers, and 
instructional coaches 

• Focus groups with teachers 

Survey data 

• Three teacher survey items* that measured the 
frequency (never, rarely, sometimes, often) of 
collaborative activities: 

1. "Consult with other teachers about challenges 1 

am facing in the classroom." 

2. "Share the content of my lesson plans with other 

teachers." 

3. "Discuss what I've learned in professional 

development activities with other teachers." 
*Although these items all measure aspects of 
teacher collaboration, they did not constitute a 
reliable scale. Thus, analysts classified schools based 
on whether schools were consistently above or 
below the mean on each. See Appendix B for further 
details. 

Culture of collaboration (4 schools) 

• Qualitative data: At least one of the following respondent groups— at least two teachers, the 
principal, or an instructional coach— reported that collaboration time is used for planning 
lessons, addressing individual student needs, or that time is otherwise described as productive; 
and at least two teachers described the working environment as "collegial"; AND 

• Survey data: School means on all three teacher collaboration items were above the overall 
sample means (3.63, 3.46, and 3.38). 

Some collaboration (17 schools) 

• Qualitative data: At least one of the following respondent groups— at least two teachers, the 
principal, or an instructional coach— reported that (1) there is formal time allotted for 
collaboration during the school day, but at least one coach or one teacher suggested that the 
time was not used productively or that it was voluntary and not well-attended; or (2) there is 
informal collaboration (e.g., teachers sharing lesson plans), but no formal time during the 
school day that is used for collaboration; AND 

• Survey data: School mean on at least one of the three teacher collaboration items was below 
the overall sample mean (3.63, 3.46, and 3.38); OR 

• Qualitative data and teacher survey data did not match (e.g., all three survey items were above 
or below each of the overall sample means [3.63, 3.46, and 3.38], but qualitative data indicated 
a "culture of collaboration" or "inconsistent collaboration"). 

Inconsistent collaboration (4 schools) 

• Qualitative data: At least one of the following respondent groups— at least two teachers, the 
principal, or an instructional coach— reported that, while there is formal time during the school 
day allocated for collaboration, it is not used consistently for collaboration; OR 

• At least two teachers reported an absence of a culture of collaboration, through the following 
examples: teachers report feeling isolated, teachers report that leadership is not supportive of 
collaboration, or teachers are "off in their own rooms doing their own things;" AND 

Survey data: School means on all three teacher collaboration items were below the overall 

sample means (3.63, 3.46, and 3.38). 
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Exhibit 7.5. Indicators of School Capacity (continued from previous page) 


Leading Indicator 

Data Sources 

Classifications 

Teacher-Teacher 
Trust (see Exhibit 
B.27) 

Survey data 

• Teacher trust survey scale 

High level of trust (3 schools) 

• Teacher trust scale average was at least 0.5 standard deviations (0.28) above the scale mean 
(2.93). 

Average or moderate level of trust (17 schools) 

• Teacher trust scale average was within 0.5 standard deviations (0.28) of the scale mean (2.93). 

Low level of trust (1 school) 

• Teacher trust scale average was at least 0.5 standard deviations (0.28) below the scale mean 
(2.93). 

Insufficient data (4 schools) 

Safe and Orderly 
Climate (see 
Exhibit B.25) 

Qualitative data 

• Interviews with district administrators, principals, 
teachers, and instructional coaches 

• Focus groups with teachers, students, and parents 

Safe and orderly (12 schools) 

• At least two respondents explicitly described the school as safe or as having none or few 
behavior problems among students, and no respondents disagreed or made statements to the 
contrary. 

Mixed (8 schools) 

• Respondents made contradicting or different statements about the school's safety and about 
student behavior. 

Unsafe and disorderly (4 schools) 

• At least two respondents described a feeling of being unsafe or behavior problems among 
students, and no respondents disagreed or made statements to the contrary. 

Insufficient data (1 school) 

Use of Data for 
Instructional 
Decisions (see 
Exhibit B.26) 

Qualitative data 

• Interviews with district administrator, principal, 
teachers, and instructional coaches 

• Teacher focus groups 

High data use (8 schools) 

• At least three respondents described using data frequently and purposefully, such as to guide 
instruction (differentiating instruction) or to identify students to pull out of classrooms or for 
after-school instruction, or to guide professional development for teachers; AND 

• Beyond reviewing student data, respondents also described using the data to guide their 
instruction (differentiating instruction), or identify students to pull out of classrooms for 
additional intervention or for Saturday schools; AND 

• School had specific people who assumed leadership for support of data use, such as 
instructional coaches, data teams, or whole-school processes. 

Medium data use (11 schools) 

• At least three respondents talked about using data to guide instruction or professional 
development for teachers. 

Low data use (6 schools) 

• Two or fewer respondents discussed reviewing student data; OR 

• Schools where the respondents specifically indicated that the school does not utilize student 
data. 
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Exhibit 7.5. Indicators of School Capacity (continued from previous page) 


Leading Indicator 

Data Sources 

Classifications 

Locus of 
Responsibility 
(see Chapter 3 or 
Exhibit B.4) 

Qualitative data 

• Interviews with the principal and teachers 

• Focus groups with teachers 

Survey data 

• One teacher survey item for which respondents 
agreed or disagreed with the following statement: 
"If teachers in this school work hard, we can meet 
our school's goals for student achievement." 

Internal responsibility (6 schools) 

• Qualitative data: Respondents described their performance problems as being within the locus 
of control of the adults in the school; or respondents described the external context as 
challenges, but assumed responsibility for the school's history of low performance and did not 
describe these challenges as insurmountable; or respondents described the external context in 
neutral terms, such as "This school is in a neighborhood with high crime"; AND 

• Survey data: School mean on the collective efficacy survey item was above the overall sample 
mean (3.07). (Two schools classified as having internal responsibility did not have adequate 
survey data, so their classification is based solely on the qualitative data.) 

Limited internal responsibility (14 schools) 

• Qualitative data: Respondents made statements that attributed the performance problems to 
the external context, but also described challenges within the locus of control of the adults in 
the school; or respondents attributed the school's history of low performance to a mix of 
internal and external factors; or respondents (i.e., at least two teachers or the principal) 
disagreed about the locus of responsibility; OR 

• Qualitative data and teacher survey data did not match (i.e., the collective efficacy item mean 
was below the overall sample mean [3.07], but interview and focus group data indicated 
"internal responsibility"; or the collective efficacy item mean was above the overall sample 
mean, but interview and focus group data indicated "external responsibility"). 

External responsibility (5 schools) 

• Qualitative data: Respondents explicitly identified factors external to the school as responsible 
for the performance problem and did not attribute the history of low performance to any 
factors internal to the school (e.g., school culture, instruction, leadership, collaboration); or 
respondents made statements that attributed the performance problems to others outside of 
the school, such as "Parent participation is why we are failing. That is the foundation"; AND 

• Survey data: School mean on the collective efficacy survey item was below the overall sample 
mean (3.07). (One school classified as having external responsibility did not have adequate 
survey data, so its classification is based solely on the qualitative data.) 


Notes: See also Chapter 2 and Appendix D for additional methodological details on the teacher survey scale measures. 
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Among the 21 of 25 core sample schools with sufficient data on all 8 leading indicators of overall 
organizational capacity, 6 were rated as having relatively higher levels of capacity, 8 as having more 
moderate capacity, and 7 as having relatively lower capacity (see Exhibit 7.6 and Box 7.3). Although 
each school fit into one of the three overall classifications, they varied in their overall levels of 
organizational capacity, with summative ratings ranging from 4 to 16. In addition, there was not a 
uniform profile for what a lower, moderate, or higher capacity school looks like. For example, some 
moderate-capacity schools tended to have moderate ratings on nearly all of the leading indicators, 
whereas other moderate-capacity schools had moderate ratings on some indicators but also high and 
low ratings on other indicators. As another example, schools with low, moderate, and high ratings on 
the 'safe and orderly climate' indicator were represented among the schools with lower overall capacity 
and with higher overall capacity. 

We now consider possible associations between overall organizational capacity and some of the other 
classifications and characteristics summarized earlier in this report. As seen in Exhibit 7.6, elementary 
schools in our core sample tend to be at the tails of our organizational capacity scale, whereas the 
high schools tend to be closer to the middle. Nonetheless, both elementary and high schools are 
represented in each of our three classifications for overall capacity. Although not displayed in Exhibit 
7.6, there was no apparent relationship between organizational capacity and the district a school is 
located in, nor an apparent relationship between organizational capacity and the particular SIG model 
that a school is implementing. 

Core sample schools with higher capacity generally were also those reporting greater access to 
material resources. For example, teachers in these schools were less likely to report having inadequate 
facilities, large classes, or too few textbooks. 40 Thus, core sample schools with measurable strengths 
with regard to leadership, climate, collaboration, and other indicators also appear to benefit from 
material assets. Schools with higher capacity were also less likely to have experienced a delay in 
receiving SIG. 

Although core sample schools with lower capacity generally were those reporting fewer material 
resources, these were also the schools that received relatively larger SIG awards, on average. 

Specifically, the five schools with the highest capacity ratings received SIG grants that were worth an 
average of 18.1 percent of 2009-10 per-pupil spending, compared with an average value of 48.6 percent 
of 2009-10 per-pupil spending among the five lowest-capacity schools. Two of the five lowest-capacity 
schools received the largest SIG awards, both in terms of the per-pupil award amount (more than 
$5,000 per pupil) and in terms of the award amount as a percentage of the prior year's per-pupil 
spending (more than 115 percent). However, all seven of the schools classified as having lower capacity 
reported experiencing a delay in receiving SIG in 2010-11, which may at least partly explain the 
apparent discrepancy between reports of fewer material resources despite receipt of relatively larger 
SIG awards. 

The lowest-capacity schools were also those in which school staff were reportedly not involved in the 
SIG application process and in which teachers reported negative perceptions of SIG. In addition, three 
of the four schools for which SIG was perceived to be the primary impetus for change (see Chapter 6) 
had lower overall capacity. 


40 Our organizational capacity measure did not incorporate the survey items about access to these material 
resources. 
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Exhibit 7.6. 

School Classifications on Leading Indicators and Overall Organizational Capacity, by Core Sample School 


School 

Leading Indicator 

Overall 

Capacity 

Rating 

Leadership 

Coherence 

Clear and 
Shared Goals 

Teacher 

Collaboration 

Teacher- 
Teacher Trust 

Safe and Orderly 
Climate 

Use of Data 

Locus of 
Responsibility 

Blizzard Bay Elementary 

Moderate 

Low 

Low 

Some 

Moderate 

Unsafe and 
disorderly 

Low 

Limited internal 

4 

Tyron Elementary 

Low 

Low 

Low 

Some 

Moderate 

Mixed 

Low 

Limited internal 

4 

Island Bay Elementary 

Low 

Low 

Low 

Some 

Moderate 

Safe and orderly 

Low 

Limited internal 

5 

Aerovista High 

Moderate 

Moderate 

Moderate 

Inconsistent 

Moderate 

Unsafe and 
disorderly 

Medium 

Limited internal 

6 

Tyro Trail Elementary 

Moderate 

Moderate 

Moderate 

Inconsistent 

Low 

Mixed 

Medium 

Limited internal 

6 

Elmsville High 

Moderate 

Moderate 

Moderate 

Inconsistent 

Moderate 

Safe and orderly 

Low 

Limited internal 

7 

Gale Secondary 

Low* 

Moderate 

Moderate 

Some 

Moderate 

Safe and orderly 

Medium 

External 

7 

Big Acorn High 

Moderate 

Moderate 

Moderate 

Inconsistent 

Moderate 

Safe and orderly 

Medium 

Limited internal 

8 

Coral High 

Moderate 

Moderate 

Moderate 

Some 

Moderate 

Safe and orderly 

Medium 

External 

8 

Inner Brooks High 

Moderate 

Low 

Moderate 

Some 

Moderate 

Mixed 

Medium 

Internal 

8 

Rossignol Elementary 

High 

Moderate 

Moderate 

Some 

Moderate 

Mixed 

Low 

Limited internal 

8 

Paul Bunyan High 

Moderate 

Moderate 

Moderate 

Culture of 
collaboration 

Moderate 

Mixed 

Medium 

Limited internal 

9 

Proctor Point High 

Moderate 

Low 

Moderate 

Some 

Moderate 

Safe and orderly 

High 

Limited internal 

9 

Sawbuck Elementary 

Moderate 

Moderate 

Moderate 

Some 

Moderate 

Mixed 

High 

Limited internal 

9 

West Marble High 

Moderate 

Low 

Moderate 

Culture of 
collaboration 

High 

Safe and orderly 

Medium 

External 

9 

Melon Elementary 

Moderate 

Moderate 

Moderate 

Culture of 
collaboration 

Moderate 

Safe and orderly 

High 

External 

10 

Meribel High 

Moderate* 

Moderate 

Moderate 

Some 

Moderate 

Safe and orderly 

High 

Limited internal 

10 

Sherbrooke Elementary 

Moderate 

High 

Moderate 

Some 

Moderate 

Mixed 

Medium 

Internal 

10 

Haven Way Elementary 

High 

Moderate 

Moderate 

Some 

Moderate 

Safe and orderly 

High 

Limited internal 

11 

Baltimore Bridge Elementary 

High 

High 

High 

Some 

High 

Unsafe and 
disorderly 

High 

Internal 

13 

Sterling Slope Elementary 

High 

High 

High 

Culture of 
collaboration 

High 

Safe and orderly 

High 

Internal 

16 


Source: SST respondent interviews and focus groups, spring 2011; SST teacher survey, spring 2011. 

Notes: Includes 21 of 25 core sample schools that had sufficient data for all eight leading indicators (see Exhibit 7.5). *Due to insufficient data on transformational and 
instructional leadership, the leadership rating for Gale Secondary and Meribel High is based on the measure of strategic leadership only. All school names are pseudonyms. 
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Core sample schools situated in traumatic external contexts appeared to have lower organizational 
capacity compared to schools in depressed or benign environments (see Exhibit 7.7 and Chapter 3). 
Four of the five lower-capacity schools were situated in traumatic environments, which may have 
constrained their ability to recruit staff, build collaborative relationships, and focus on other school 
improvement efforts. That said, there was one school with higher overall capacity despite being situated 
in a traumatic environment, and there were two schools with lower overall capacity despite being 
situated in a benign environment. 


Exhibit 7.7. 

Overall Organizational Capacity of Core Sample Schools, by Perceived External Context 



Higher Overall Capacity 
(ratings 10-16) 

Moderate Overall 
Capacity 
(ratings 8-9) 

Lower Overall Capacity 
(ratings 4-7) 

Total 

Benign 

External 

Context 

2 schools 
Meribel High 
Sherbrooke Elementary 

3 schools 
Coral High 
Paul Bunyan High 
Sawbuck Elementary 

2 schools 
Elmsville High 
Gale Secondary 

7 schools 

Depressed 

External 

Context 

3 schools 

Baltimore Bridge Elementary 
Haven Way Elementary 
Melon Elementary 

5 schools 
Big Acorn High 
Inner Brooks High 
Proctor Point High 
Rossignol Elementary 
West Marble High 

1 school 
Aerovista High 

9 schools 

Traumatic 

External 

Context 

1 school 

Sterling Slope Elementary 


4 schools 

Blizzard Bay Elementary 
Island Bay Elementary 
Tyron Elementary 
Tyro Trail Elementary 

5 schools 

Total 

6 schools 

8 schools 

7 schools 



Source: SST respondent interviews and focus groups, spring 2011. 

Notes: Includes 21 of 25 core sample schools that had sufficient data to determine overall organizational capacity. All school 
names are pseudonyms. 


School Capacity and Perceived Improvement 

Core sample schools with higher overall organizational capacity appeared to be more likely to report 
improvement in many areas in 2010-11 (see Exhibit 7.8). Four of the six schools with higher overall 
capacity reported improvement in many areas in 2010-11, while four of the eight schools with 
moderate overall capacity reported improvement in some areas, and six of the seven schools with lower 
overall capacity reported improvement in few or no areas. 

There were exceptions to this general trend, however. For example, respondents at Melon Elementary 
reported improvement in few areas despite higher ratings of capacity. One potential explanation is that 
this school may have had higher capacity prior to SIG, but we are unable to confirm that this was the 
case because we do not have a measure of capacity in 2009-10. At this school, teachers and 
administrators described changes prior to SIG, with perceived improvement in years prior to, but not in, 
2010-11. Thus, reports of improvement in 2010-11 may have been less prevalent because staff 
perceived a steady, "stay-the-course" approach rather than actions that would prompt visible 
improvements. 
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Exhibit 7.8. 

Overall Organizational Capacity of Core Sample Schools, 
by Level of Perceived Improvement in 2010-11 



Higher Overall Capacity 
(ratings 10-16) 

Moderate Overall 
Capacity 
(ratings 8-9) 

Lower Overall Capacity 
(ratings 4-7) 

Total 

Schools With 
Reports of 
Improvement 
in Many Areas 

4 schools 

Baltimore Bridge Elementary 
Haven Way Elementary 
Meribel High 

Sterling Slope Elementary 

2 schools 
Inner Brooks High 
West Marble High 


6 schools 

Schools With 
Reports of 
Improvement 
in Some Areas 

1 school 

Sherbrooke Elementary 

4 schools 
Big Acorn High 
Coral High 

Rossignol Elementary 
Sawbuck Elementary 

1 school 

Blizzard Bay Elementary 

6 schools 

Schools With 
Reports of 
Improvement 
in Few or No 
Areas 

1 school 

Melon Elementary 

2 schools 
Paul Bunyan High 
Proctor Point High 

6 schools 
Aerovista High 
Elmsville High 
Gale Secondary 
Island Bay Elementary 
Tyron Elementary 
Tyro Trail Elementary 

9 schools 

Total 

6 schools 

8 schools 

7 schools 



Source: SST respondent interviews and focus groups, spring 2011. 

Notes: Includes 21 of 25 core sample schools that had sufficient data to determine overall organizational capacity. All school 
names are pseudonyms. 


As another example, respondents from Blizzard Bay Elementary described improvement in five areas 
despite having the lowest rating of overall capacity among our core sample schools. Respondents from 
Blizzard Bay reported having a challenging student population, and the principal reported that teachers 
lacked the skills to address student needs. The students are mobile and from disadvantaged 
backgrounds. Many are recent immigrants from war-torn countries with no prior educational 
experience. Because of SIG, the principal was replaced in 2010-11, with the new one reportedly being 
highly regarded by teachers, district officials, the union representatives, and parents. Overall, 
respondents suggested that Blizzard Bay had improved with regard to leadership, staff expectations for 
students, collaboration, instruction, and material resources, yet it was still rated as "low" with regard to 
capacity for coherence, data use, clear and shared goals, and school climate. Although the principal 
expressed confidence in his improvement plan, he noted that changing school culture and instruction 
would take time. Thus, although respondents described improvement, our other data suggest they still 
may have work to do. 

All six lower-capacity schools that did not report at least some improvement in 2010-11, despite SIG 
intervention and financial support, faced challenges described earlier in the report: a lack of strategic or 
transformational leadership, untimely principal turnover, a particularly challenging context, a SIG 
application process that excluded teachers, a divisive culture, a delay in SIG funds, or a combination of 
these. 
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However, our findings are not intended to suggest that SIG succeeded only in schools where there was 
already capacity for improvement. Indeed, we should note some caveats with regard to the 
interpretation of these data. Most notably, the thresholds between the classifications (for example, 
high, moderate, and lower capacity) are somewhat arbitrary, and if the thresholds shifted slightly, 
schools could move from one classification to another. In addition, the measures of perceived 
improvement and organizational capacity are concurrent, both collected at the end of the 2010-11 
school year. Although this study is not designed to permit causal inferences, sequential measures could 
have bolstered confidence in this association. But given the timing of our data collection in Year 1 of SIG, 
we are only able to report concurrent measures. 

Chapter Summary 

Chapter 3 illustrated that while all of our 25 core sample schools are persistently low-performing, they 
differ with regard to their external context, financial resources, and performance problems. Chapter 4 
illustrated the variation with regard to leadership capacity, and Chapter 5 described the assorted 
improvement actions implemented. In Chapter 6, we described whether schools experienced a 
disruption from the past and the extent to which this break was associated with SIG. In addition, we 
depicted the variation with regard to involvement of school staff in the SIG application process, delays in 
funding, and monitoring and support from districts and states. 

Given this variation in histories, context, and improvement actions, it is perhaps not surprising that at 
the end of the first year of SIG, some schools appeared to be better positioned than others to continue 
their improvement efforts. The core sample schools appeared to differ substantially in their overall 
organizational capacity. Three of the four schools in which SIG was perceived to be the primary impetus 
for change were also rated as having the lowest overall capacity and were ones that received relatively 
larger per-pupil SIG awards. However, respondents in the lowest-capacity schools did not always report 
corresponding improvements. All but one lower-capacity school perceived improvement in only a few or 
no areas in 2010-11. On the other hand, all but one higher-capacity school reported improvement in 
some or many areas in 2010-11. 

The patterns described in this chapter may change in subsequent years, as 2010-11 was just the first 
year of a three-year grant, and as we saw in Chapter 6, many of our core sample schools were off to a 
late start with SIG due in part to funding delays. 
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Conclusions 


The SIG program, first authorized in 2001, provides formula-based federal funds to states that then 
competitively award these funds to districts applying for SIG on behalf of their persistently lowest- 
achieving schools. These schools use the funds to implement reforms intended to turn around a history 
of low performance. With the passage of the American Recovery and Reinvestment Act of 2009 (ARRA), 
SIG funding was increased by about 6.5 times, and SIG design and requirements revamped. These 
modifications were designed to better target SIG to the nation's lowest-achieving schools and to ensure 
that more aggressive improvement strategies are adopted for such schools than had been previously. 
Since the passage of ARRA, four cohorts of schools have received SIG as of the 2013-14 school year. 
Cohort I grantees include schools that received SIG during the fiscal year 2009 competition cycle to 
implement reforms beginning in the 2010-11 school year. Cohort II grantees include schools that 
received SIG during the fiscal year 2010 competition cycle to implement reforms beginning in the 2011- 
12 school year. Cohorts III and IV were awarded for the 2012-13 and 2013-14 years. This report has 
focused on the actions of a purposive sample of Cohort I SIG recipients in the first year of 
implementation during the 2010-11 school year. 

The preceding chapters have highlighted the complexity of the schools themselves and the process of change 
in which they were engaged during 2010-11. In general, we find that the role of the SIG program differs 
across the core sample schools, and that these schools appear to be in different positions for moving ahead 
after the first year of the grant. In Chapter 1, we began by stating that this report was to provide an initial 
picture of: (1) the schools and what they are doing, and (2) the change process in these schools. 

About the Schools and What They are Doing 

Although all of our core sample schools were low performing, they differed in their community and fiscal 
contexts, their performance and reform histories, and their interpretations of the causes of - and 
potential solutions for - their performance problems. Our core sample schools were situated in a range 
of community contexts, from "traumatic" neighborhoods characterized by reports of high crime, 
incarceration, abuse, and severe urban poverty to comparatively "benign" environments in which 
limited crime was reported, homes were in good repair, and there were few reports of family instability. 
Our core sample schools also varied in the degree to which they attributed their school's performance 
problem to factors within their control (internal causes) or outside of their control (external causes). In 
the six core sample schools that appeared to accept internal responsibility for their performance 
challenges, stakeholders appeared to be more committed to working to improve the school despite the 
challenges that they faced. 

As required by SIG, the majority of our core sample schools (21 out of 25) replaced their principals in 
either 2009-10 or 2010-11 (one school did so twice). Approaches to leadership varied across the core 
sample schools, with most principals exhibiting a mix of leadership qualities. When classified based on 
three dimensions of leadership (transformational, instructional, and strategic), few principals (2 of 25) 
placed high and few principals placed low (2 of 25) on all three dimensions. The majority of principals 
(21 of 25) reportedly exhibited a mixture of these qualities. 

Core sample schools reported engaging in a wide range of improvement actions and uses of funds. The 
most frequently reported improvement actions included expanding professional development, 
increasing learning time, or replacing the principal. Although SIG schools must adhere to SIG program 
mandates, school respondents reported varying rationales for selecting improvement strategies and 
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actions, as well as varying perceptions of how the selected improvement strategies and actions fit into 
their overall change process. Indeed, we found that the SIG models do not always necessarily dictate 
what actions the schools initiate. For example, of the nine core sample schools that reported replacing 
at least 50 percent of teachers, three were implementing the turnaround model, for which this action is 
required, while the remaining six were implementing the transformation or restart models, for which 
this action is not required. 

That our core sample schools were using SIG funds to implement varying improvement strategies should 
not be surprising, given the diversity across the schools and the fact that many had been the subject of 
improvement initiatives and accountability policies over the years. The schools were thus not "blank 
slates" for reformers to craft anew. Rather, they were existing organizations with prior reform histories 
in which the participants tried to leverage change by addressing identified performance problems, as 
well as implementing specific requirements of the SIG program. 

About the Change Process in These Schools 

Grounded in literature on school turnaround, the SIG program's requirements reflect a hypothesis that 
the lowest-performing schools require a disruption from the past. We found that 7 of our 25 core 
sample schools had experienced a "visible disruption" from past practices, while the remaining schools 
appeared to be following a more incremental approach to improvement. SIG did not appear to be the 
primary impetus for change in most of our core sample schools. In 19 of our 25 schools, the overall 
approach to school improvement in the first year of SIG (2010-11) was reportedly a continuation of 
activities or plans that predated SIG. 

Initial implementation of schools' selected improvement actions was, in some cases, reportedly 
hampered by issues such as delays in the distribution of SIG awards. In addition, support for some 
schools was reportedly limited due to inadequate capacity at both the state and district levels. Despite 
these limitations, it is still somewhat surprising that in two of our core sample schools, SIG 
implementation was characterized by respondents as "business-as-usual." 

At the end of the first year of SIG, some of our core sample schools appeared to be in a better position than 
others to improve and sustain student achievement. Although the majority of our schools reported some 
improvement in 2010-11, schools with reportedly higher levels of principal strategic leadership and that had 
experienced a visible disruption from past practices tended to report more improvement. In addition, the 
core sample schools varied with regard to their organizational capacity. The schools with lower capacity 
tended to be in traumatic contexts and report lower levels of resources on the teacher survey. 

Upcoming: The Change Process in Years 2 and 3 of SIG 

This report documents SST's findings based on one year of data, which reflects only the first of three 
years of SIG funding. The individuals, actions, challenges, and accomplishments reported in Year 1 may 
or may not remain in subsequent years. Schools that reported improvement in Year 1 may falter, while 
others may be "late bloomers," in which improvement processes need more time to take hold. To 
document the continuing change process, we will release one more report tracking these persistently 
low-performing schools in Years 2 and 3 of SIG. We will continue to follow their improvement efforts 
and document the role of SIG in their struggles to turn around a history of low performance. 
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Appendix A. Study of School Turnaround Codebook 

Appendix A provides the codebook developed to assist analysts as they sorted and labeled data from interview and focus group 
transcripts. Individual codes were applied to either the entire transcript or segments of data within transcripts so that these coded segments 
could be retrieved and sorted efficiently to facilitate the study team's secondary analyses. The codebook included a list of all codes sorted by 
category, along with a definition for each category and each individual code. Examples of types of data that fell into each code category were 
listed when necessary, as were exclusions. The complete set of data collection instruments can be found at 
http://www.air.org/topic/education/study-of-school-turnaround-year-one-protocol-survey. 

SCHOOL BACKGROUND (!): Codes in this category apply to the full interview or focus group. Attach these codes to the top of the document 
once. 


Code 

Definition 

Examples 

Exclusions 

(Elementary 

Interview or focus group code referring to the 
type or level of school. 

Schools that serve grades K-8 or any derivation of 
these grades. 

District files are not to be labeled 
with this code. 

IHigh School 

Interview or focus group code referring to the 
type or level of school. 

Schools that serve grades 9-12 or any derivation of 
these grades. 

District files are not to be labeled 
with this code. 

I Restart 

Interview or focus group code that refers to 
the SIG restart model. 

As categorized by SIG subgrant application. 

District files are not to be labeled 
with this code. 

IRural 

Interview or focus group code that refers to 
schools and districts in rural area. 

As categorized in 2008-09 Common Core of Data. 


ITransformation 

Interview or focus group code that refers to 
the SIG transformation model. 

As categorized by SIG subgrant application. 

District files are not to be labeled 
with this code. 

(Turnaround 

Interview or focus group code that refers to 
the SIG turnaround model. 

As categorized by SIG subgrant application. 

District files are not to be labeled 
with this code. 

lUrban 

Interview or focus group code that refers to 
schools in urban areas (including urban 
fringe). 

As categorized in 2008-09 Common Core of Data. 
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PARTICIPANT BACKGROUND (#): Codes in this category apply to segments of data related to the background of the participants being 
interviewed or participating in the focus groups. Apply these codes to segments of data that outline individual participants' histories at the 
school or district. 


Code 

Definition 

Examples 

Exclusions 

#long history at school 

Participant has been working at the specific 
case study school for 10 or more years. 



#new to district 

Participant is in their first or second year being 
employed by this specific school district. This 
means that they assumed this position at any 
time during the 2009-10 or 2010-11 academic 
years, including the summer prior to school 
starting. They have had NO prior experience in 
the district in any other positions. 

New teacher, new principal, new district personnel. 

If a participant worked in the 
district for some period of time, 
left, and then returned, they are 
not considered #new to district. 

#new to role 

Participant has no prior experience serving in 
their current role/job. 

First year teacher, first year principal, first year 
coach. 

If a participant has worked in 
DIFFERENT schools or districts 
doing the same job, they are not 
considered #new to role. 

#new to school 

Participant started working at the school as of 
the 2009-10 or 2010-11 academic years. If a 
participant has started anytime during those 
years, they are considered #new to school. 

Teacher, principal, external provider 

If also new to district or for district 
staff, use # new to district. 
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MULTIPURPOSE CODES (*): Codes in this category apply to segments of data from individual interviews and focus groups. These codes are used 
primarily in conjunction with codes from other categories to specify specific stakeholder groups involved in activities or constructs being 
discussed (e.g., school leader/district leader/teacher) or capture constructs that fall outside of the remaining categories. There is no school 
multipurpose code because the school is the unit of analysis and assumed to be the default by study participants. 


Code 

Definition 

Examples 

Exclusions 

■"Capacity 

Data that reference the extent to which different staff 
members at the school or district have the capacity to carry 
out their respective roles. Includes evidence along the entire 
capacity spectrum from low capacity to high capacity. 

Leadership capacity, teaching 
capacity. 

Related to perceived expertise but 
different in that this code can 
apply to individuals or 
organizations. 

If data is related to a leader's 
ability to lead effectively, consider 
double-coding with Leadership. 

■"Challenge 

Data that reference issues, concerns or conditions that 
study participants perceive as challenges. 

Low levels of parent participation, 
low-capacity leaders or teachers, lack 
of resources at school or district level. 

MUST be identified by study 
participant, NOT researcher's 
opinion. 

■"Change Strategy 

Data that reference overarching school or districtwide 
change strategies meant to "turnaround" low-performing 
schools, or more targeted change actions and strategies 
focused on a specific area of reform (e.g., curriculum or data 
use). 

Curriculum, data use, student 
supports, change school culture, 
improve instruction. 


■"Community 

Data that reference the community role in the school or 
district. 

Characteristics of school community, 
community involvement in school or 
district activities. 


■"District 

Data that reference personnel, policies or practices taking 
place at the district level that are relevant to school change; 
identify school-level policies, practices, or conditions that 
are influenced/impacted by the district; or identify 
contextual issues that are district specific. 

District-level change strategies, 
context, SIG processes, rationales or 
problem definitions; district capacity. 

Do not use to identify a participant 
that works for the district. 

■"District leader 

Data that reference high-level district leaders, such as 
superintendent or director of elementary education. Used in 
addition to other codes on the list for segments of data that 
are specific to the district leader (personal background of 
district staff), or reflect a perspective on district leadership 
(either by teachers, principal, parents, students or other 
school staff). 

Role definition, capacity, problem 
definition. 


■"External support provider 

Data that reference external support provider roles and 
responsibilities, capacity to support schools, and 
perceptions of effectiveness or usefulness of these providers 
by other school stakeholders. 

Role definition, capacity, morale, 
perceived expertise. 
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Code 

Definition 

Examples 

Exclusions 

*lnstructional coach 

Data that reference instructional coach roles and 
responsibilities, capacity to support schools, and 
perceptions of effectiveness or usefulness of these coaches 
by other school stakeholders. 

Role definition, coach background, 
coach capacity, coach morale, coach 
buy-in, staffing, data use, 
collaboration, rationale. 


*Morale 

Data that reference morale at case study schools. Morale 
can cross stakeholder groups and apply to teachers, parents, 
students and even school leaders. This code references the 
whole spectrum of morale levels from low to high. Related 
to buy-in in that data describing instances of low morale 
among staff, students, parents or the community may also 
signal instances of low buy-in. 



*Other funding 

Data that reference activities taking place at the school or 
district that are not funded by SIG. 

Title 1; State improvement grants; 
grants for EL or Special Education. 


*Other School Staff 

Data that reference other school staff (e.g., social worker, 
counselor) roles and responsibilities, capacity to support 
schools, and perceptions of effectiveness or usefulness of 
these staff by other school stakeholders. 

Efficacy of school counselors, role of 
school social workers or safety 
attendants. 


*Parent 

Data that reference how parents are involved at the school. 
Can be used in conjunction with codes for engagement, 
commitment to the school, morale or buy-in. 

Commitment to school, commitment 
to students, culture, priorities, buy-in, 
morale. 


*Perceived expertise 

Data that reference the extent to which specific school or 
district staff have the background, training, and knowledge 
that are necessary and appropriate for them to do their jobs 
successfully. Can be a respondent's own perceptions of their 
expertise or someone else's perceptions of them. These 
perceptions must be explicitly discussed by study 
participants, not inferred by analysts based on data. 

Past leadership experience of 
principal, extent to which literacy and 
math coaches have received training 
to coach other teachers in these 
content areas. 

Related to capacity but different in 
that this code captures 
perceptions about whether a 
person's skills (e.g., literacy coach) 
are sufficient for them to carry out 
their job responsibilities. 

*Planned activities 

Data that reference school, district, or state activities that 
are planned but will be implemented in future years. 

Evaluation systems, professional 
developments, new curriculum. 

For activities that are planned and 
will be funded by SIG use 
SIG_Planned activities. 

*Priorities 

Data that reference state, district, or school-level priorities. 
These may or may not align with SIG activities and may or 
may not be considered part of a school's problem definition. 

Raising test scores, improving 
behavior, attendance, parent 
involvement, improved capacity, 
replacement of teachers. 
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Code 

Definition 

Examples 

Exclusions 

*Problem definition 

Data that reference what study participants identify as the 
key leverage points or targets for moving the school 
forward. Data that capture participants' perspectives on 
what needs to change in order for their schools to improve. 
Will be double-coded with challenge. 

Staff capacity, school culture, student 
behavior. 

Issues that participants identify as 
challenges, but that are not key 
issues that need to be addressed 
to move school forward should be 
coded with *Challenge only (when 
identified as a challenge by the 
study participant). 

*Rationale 

Data that reference stakeholders' rationales for making 
choices about SIG and/or any other decisions being made at 
the school or district levels. This includes respondents' 
descriptions of their own rationales, but also respondents' 
descriptions of others' rationales. 

Rationales about staffing, curricula, 
SIG applications, SIG model selection, 
data use, etc. 


*Role definition 

Data that reference how study participants 
perceive/describe their job role and/or responsibilities. Can 
also include descriptions of how other stakeholders perceive 
the roles of their colleagues/peers. For example, this code 
could be used for a description of how an instructional 
coach perceives their job role, but also a description of how 
a principal perceives the job role of the same coach. 

Job role, duties, responsibilities, 
differences in expectation of role 
versus reality of job. 


*School leader 

Data that reference other school leaders (e.g., assistant 
principal, leadership team) roles and responsibilities, 
capacity to support schools, and perceptions of 
effectiveness or usefulness of these staff by other school 
stakeholders. 

Role definition, capacity. 

Not used to convey that a specific 
opinion was expressed by this 
study participant. For example, if a 
principal is describing why they 
think their school is low 
performing, you would not code 
that passage as *school leader 
because the principal is speaking. 

*SIG-funded 

Data that reference activities taking place at the school or 
district that are being funding by SIG. 

PD, new hires, new technology, new 
instructional materials, extra 
instructional time, new curricula. 

Activities where funding source 
wasn't specifically identified as 
SIG. 

*State 

Data that reference state officials roles and responsibilities 
with respect to SIG support or district support and 
perceptions of effectiveness or usefulness of these staff by 
other school stakeholders. State policies or practices that 
are influencing SIG schools. 

State support for SIG schools or 
districts, state restrictions on use of 
SIG funds, state processes that impact 
distribution of SIG funding, state role 
in renewal of SIG year to year. 

Not used simply to identify a 
participant that works for the 
state. 

*Strength 

Data that references something that is considered a 
strength for either themselves personally or the school or 
district as organizations. 


MUST be identified by study 
participant, NOT researcher's 
opinion. 
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Code 

Definition 

Examples 

Exclusions 

*Student 

Data that reference perspectives on students in a district or 
school. 

Morale, buy-in, academic 
performance, commitment to school, 
engagement. 


*Teacher 

Data that reference perspectives on teachers in a district or 
school in addition to teachers' roles and responsibilities. 

Role definition, teacher background, 
teacher capacity, teacher morale, 
teacher buy-in, staffing, data use, 
collaboration. 

Focus on the content of the 
quotation and not the person 
speaking. Teacher code should not 
be used simply to attribute a 
comment to a teacher. 

*Unclear funding 

Data that reference activities taking place at the school or 
district where a specific funding source cannot be identified. 

Only use if the respondent is explicit 
about the fact that they don't know 
where the funding for a specific 
activity is coming from. 


*Union 

Data that reference the role of the union at the district or 
school level. Also can be used in conjunction with other 
codes to capture content related to union politics, union 
role in staffing, SIG processes. 

Union contracts, union politics. 
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DIMENSION CODES (%): Codes in this category apply to segments of data that address key first-year dimensions that are more analytic and less 
descriptive. These three codes can be applied to data from all stakeholder groups at the state, district, and school levels. These three codes are 
used to "tag" data related to buy-in, divergence, and coherence. 


Code 

Definition 

Examples 

Exclusions 

%Buy-ln 

Data that reference the extent to which teachers, 
administrators, parents, or students are supportive of 
various reform efforts or changes taking place at their 
school. This code reflects data across the entire spectrum of 
the construct, from evidence of low levels of buy-in through 
very high levels. 

Teachers support new curriculum and 
are implementing it with fidelity; 
parents support new school leader 
and are making an effort to introduce 
leader to larger school community. 


%Coherence 

Data that reference the extent to which stakeholders 
perceive reform efforts to fit together and be in service of 
school improvement efforts overall. 



%Divergence 

Data that reference the extent to which activities are similar 
or different from activities that the school has implemented 
in the past. Determining similarity can involve issues of 
intensity or newness. For example, if a school is continuing 
an existing reform but in a far more intense manner, this 
could be considered divergence. Likewise, if a school 
implements an entirely new reform this is also divergence. 
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SIG PROCESSES (SIG): Codes in this category apply to segments of data that describe how SIG schools are navigating the SIG process. These 
codes can describe aspects of SIG at any level of the system (state, district, school) and should be paired with the appropriate multipurpose code 
to separate district-level from school-level data. 


Code 

Definition 

Examples 

Exclusions 

SIG_Application Process 

Data that reference what the SIG application process was 
like. Could include who was involved in the application 
process and perspectives on the benefits and drawbacks of 
the process. 

Timelines for drafting application or 
submission of application to district or 
state. 


SIG_Distribution of funds 

Data that reference how and when SIG funding reached 
schools. This code also includes data on any policies or 
procedures at the state or district levels that impacted the 
funding distribution process or any district level data on 
how many schools received SIG funds and how those 
decisions were made. 

Funding distribution policies at the 
state or district levels; timelines for 
funding distribution and how these 
timelines met preliminary 
expectations. 


SIG_Funding restrictions 

Data that reference restrictions in place that limit how 
schools can use SIG funds. 

Things district officials said they could 
not include in SIG, or things principals 
say they are required to use money 
for because of how the grant was 
written. 


SIG_Grant revisions 

Data that reference desired or proposed revisions to SIG for 
future funding cycles. 

Descriptions of the school revising the 
SIG budget and resubmitting for 
approval, or descriptions of something 
they wanted to do in Year 1 but could 
not due to delayed funding. 


SIGJmplementation 

Data that reference how SIG was implemented at the school 
or district level and focus on timelines for implementation 
and/or who was responsible for different aspects of 
implementation. Data that discuss the progress of 
implementation specifically would also fall under this code, 
as would data that specifically identify challenges that 
schools or districts have faced during early implementation 
of SIG (double code with challenge code). 

Who was responsible for 
implementing a certain strategy, how 
the district supported a school in 
implementation. 

Do not include data that simply 
identify SIG-funded activities. The 
data must specifically discuss the 
implementation of these change 
strategies. 

SIG_MISC 

Data that reference other SIG processes that are not 
captured by existing SIG codes. 



SIG_Model Selection 

Data that reference how and why district and/or school 
leaders selected specific SIG turnaround models during the 
SIG application process. 

Fit with existing reform efforts; 
number of eligible district schools; 
existing staffing concerns. 
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Code 

Definition 

Examples 

Exclusions 

SIG_Oversight 

Data that reference who oversees SIG, descriptions of 
monitoring efforts. Any mentions of compliance can be 
coded as oversight. 

District staff oversee groups of SIG 
schools; external partners monitor 
implementation of SIG. 


SIG_Perceptions of SIG 

Data that reference how various school stakeholders 
perceive SIG. Could include positive or negative perceptions 
about usefulness of SIG at the school, SIG's ability to 
improve student performance, or any other identified 
school problems. Can also be used if a study participant is 
not aware of SIG. 

Low impact; SIG doesn't meet needs 
of the school; SIG has allowed school 
to make positive changes to school 
staff. 


SIG_Planning 

Data that reference the planning of SIG activities. These 
could include planning during the application process, 
school- or district-level planning related to SIG, and data 
about future change strategies that are being funded by SIG 
but only in the planning phase. 

Planning for extended instructional 
time that hasn't yet begun, planning 
for Year 2 of SIG. 


SIG_Support 

Data that reference any support provided to teachers, 
schools, and districts to implement SIG. 

Technical assistance, specific state or 
district positions created to support 
SIG schools, additional help with 
budgets or exemptions from specific 
district policies. 

If something was intended as 
support but perceived as oversight 
use the SIG_Oversight code. 
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DOMAINS OF ACTIVITY ($): Codes in this category apply to segments of data that describe specific strategies, tactics, or activities that schools 
are continuing to use or have begun using during the 2010-2011 school year. These strategies can be funded through SIG or through other 
funding streams. The codes are to aligned with the study's conceptual framework. The codes are intended to be broad, so they should include all 
data that touches on the topic identified by the code. Also, the multipurpose codes (especially change strategy) will be used in conjunction with 
these domain codes to identify specific stakeholder groups or system-level actors involved with these activities. 


Code 

Definition 

Examples 

Exclusions 

$Collaboration 

Data that reference professional learning opportunities that 
focus on building collaboration between school stakeholders. 
This could be formal or more informal. It may be between 
only teachers, but could also be between teachers and 
administrators. Structured opportunities for common 
planning time for teachers either within or across grade level 
or subject. 

Department team meetings, school 
leadership teams, common planning 
time. 


$Curriculum 

Data that reference activities related to school curriculum. 
Includes existing or new curriculum, as well as 
modifications/enhancement of existing curriculum. Includes 
aligning new or existing curriculum to district, state, or 
national learning standards. Double-code with multipurpose 
codes related to funding source, rationale, professional 
development and level (e.g., district or school) when 
applicable. 

New or current math, ELA, or ELL 
curriculum. 


$Data type 

Data that reference type of data being collected or analyzed. 

State assessments, district 
assessments, school assessments, 
student work. 


$Data Use 

Data that reference how data are being used by 
administrators, staff, and external stakeholders. 
Communication and interpretation of data, level of data 
literacy, frequency of data use to drive decisions and modify 
instruction. 



$Engagement 

Data that reference strategies that address 
involvement/participation at the school. Can be directed at 
parents, students or community. Also addresses overall levels 
of engagement at school— both current or historical. 
Perceptions of engagement or efforts at outreach in general. 

Hiring a parent liaison, hiring 
bilingual staff, monthly "coffee 
chats" with the principal, parent 
well-being classes, computer 
tutorials, English classes, 
community outreach, history of low 
community or parent 
engagement/involvement. 
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Code 

Definition 

Examples 

Exclusions 

$Evaluation Systems 

Data that reference activities related to school or district 
evaluation systems. Includes new and existing evaluation 
systems, as well as changes to existing systems. 

New review practices for principals, 
teachers, schools; different people 
involved in review, different 
measures of success or failure. 

Does not include student-level 
evaluations or assessments; 
although, these may be tied to 
evaluation outcomes. 

$Governance 

Data that reference the management structure of a school. 
Think of these descriptions as formal structures. This could be 
site-based management, structures related to a restart model 
like CMO or EMO. Descriptions of how school governance 
models are working, changes to models in recent years, etc. 



$lncentives 

Data that reference programs that create incentives for 
achieving desired outcomes. Can include existing incentive 
programs. Can also include other strategies to recruit and/or 
retain staff such as a "bonus" for hard to staff schools. 
Double-code with multipurpose codes to specify whether 
incentives are aimed at teachers, students, or administrators, 
and whether they are part of SIG or not. 

Monetary bonuses for teachers or 
principals linked to student test 
scores. Incentives for teaching at 
hard to staff schools. Rewards for 
students for good behavior, 
attendance, etc. 


$lnstruction 

Data that reference instructional strategies. Instructional 
strategies focus on how teachers are teaching students, as 
opposed to the content of the lesson (the curriculum). 
Double-code with descriptive codes related to funding, 
rationale, or professional development when applicable. 

SDAIE, differentiated instruction, 
intervention for struggling students, 
computer-based instruction. 

Strategies that are not focused 
directly on classroom 
instruction. Do not double- 
code with Teacher. 

$Leadership 

Data that reference district, school, teacher, student, or 
parent leadership. Could be related to a specific leader's style 
(authoritarian) or about specific leadership opportunities that 
exist at the school. Can also address issues of ineffective or 
effective leadership, leadership changes being made at 
school or district levels, stakeholders' perceptions of their 
school and district leaders. 


If data is related to a specific 
leader's leadership abilities, 
consider double-coding with 
Capacity. 

$Professional Development 

Data that reference any type of professional development 
offered by the state, district, or school. May also be 
associated with building capacity. In such cases, double-code 
with the capacity multipurpose code. Includes frequency of 
PD, who provides PD, who attends PD, perceived usefulness 
of PD, support for implementation of PD, and content of PD. 

PD series or one-time events, off- 
site training or on-site training, 
workshops, structured classroom 
observations, conferences. 

Do not double-code with 
Teacher. 
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Code 

Definition 

Examples 

Exclusions 

$Staffing 

Data that reference staffing decisions or conditions. Can be at 
any level— state, district, school. Includes changes in 
administrators, teaching staff, classified staff, teaching 
assistants, substitutes, instructional coaches, etc. Can include 
addition of new staff and/or changing roles and 
responsibilities. Also includes where staff were hired from if 
they were added to the teaching staff (e.g., from district 
layoff list or outside of district). Double-code with 
multipurpose codes to add detail. 

Replacing existing teachers with 
different ones (this could have 
occurred because of SIG model or it 
can be something that is planned); 
hiring custodians, cafeteria staff, 
office staff; adding coaching 
positions. Changes/additions to 
principals or other administrative 
staff. 


$Student Supports 

Data that reference programs and services that support 
students but operated outside of direct classroom 
instructional time. 

Social-emotional supports; school 
counselors; medical assistance 
programs. 

Code academic after-school 
activities as Use of time. 

$Technology 

Data that reference technology use. Can describe technology 
that has been purchased or distributed as part of a school's 
improvement efforts, perspectives on technology use within 
school or district, overall strategies related to technology use. 
Double-code with descriptive codes when applicable. 

New computers, smart boards for 
classrooms, laptops for teachers, 
overall strategies related to 
technology use, perspectives on 
technology use within school or 
district, new software. 


$Use of time 

Data that reference strategies for extending, shortening, or 
restructuring learning time for students, strategies for 
restructuring work time for staff, amount of time being 
added, subtracted, or restructured, and the stated rationales 
for their use. Includes academic after-school activities like 
tutoring programs. 

Saturday school, Academic after- 
school activities, block scheduling, 
extra class periods. 

Code non-academic after- 
school activities as student 
supports. 
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CONTEXT (C): Codes in this category apply to segments of data that reference aspects of school context including, but not limited to, past or 
present reform efforts, school history, academic performance and student demographics. 


Code 

Definition 

Examples 

Exclusions 

C_Academic performance 

Data that reference organization level 
comments about the academic performance 
at the school or district generally. Can be this 
academic year or historical (e.g., history of low 
performance). 

Low academic performance, changes in academic 
performance levels over time, patterns of 
performance among different groups of students, 
achievement gaps, reasons for high or low levels of 
academic performance. 

Not used to talk about specific 
students. 

C_Attendance 

Data that reference attendance of students 
and staff in the school. Could be challenges 
like high student or teacher absenteeism or 
the opposite. 



C_Behavior 

Data that reference current or past trends in 
behavior of students as a whole. 


For specific data related to actions 
or strategies designed to improve 
student behavior, use the student 
supports code. 

C_Commitment to school 

Data that reference the extent to which 
respondents express commitment to the 
school themselves or describe perceptions of 
other people's commitment to the school. 
Double-code with parent, teacher, student, 
etc. depending on the stakeholder group that 
is being described in the data. 



C_Commitment to students 

Data that reference the extent to which 
respondents express commitment to the 
students themselves or describe perceptions 
of other people's commitment to the 
students. Double-code with parent, teacher, 
principal, etc. depending on the stakeholder 
group that is being described in the data. 



C_Community relations 

Data that reference the relationship among 
community members. This includes the 
relationships between community members, 
community members and students, community 
members and the school/school staff. 

The relationship (good or bad) between community 
members and the principal and/or school. The 
relationship between two separate groups or 
different neighborhoods within the same 
community. 

For data about racial relations, use 
C_Race relations. Does not need 
to be double-coded with C_Race 
relations. 

C_Culture 

Data that reference the type of culture that 
exists at the school or district and whether the 
school culture has changed over time and 
why. 

Positive, negative, supportive, collaborative, 
welcoming. 
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Code 

Definition 

Examples 

Exclusions 

C_Facilities 

Data that reference the condition of school 
facilities or changes that have been made that 
relate to school facilities (like moving to a new 
facility). 

Facility improvements (e.g., painting, improved 
bathrooms, technology wiring) or facility deficits 
(problems with the school facility). 


C_Level of funding 

Data that reference the overall amounts of 
money reaching the school or district. Data 
could relate to sufficiency of funds, changes to 
funding levels over time, various funding 
streams and the levels of funding associated 
with each one. 

Budget cuts, chronically low or high funding, 
additional grants or state programs that are 
increasing a school's overall funding. 

Not meant to capture SIG specific 
funding unless data refer to how 
SIG fits in with the overall level of 
funding for the school. 

C_Local politics 

Data that reference local politics that impact 
the school context. 

This may relate to politics at the district, school 
board or community level that impact the school. 

This does not refer to state 
politics. 

C_Location 

Data that reference school or district 
geography. Defining characteristics of a school 
or district that are related to its 
geography/location. 

Isolated, surrounded by public housing, district very 
spread out. 


C_MISC 

Data that reference school context issues that 
are not adequately covered through other 
codes, but describe an element of school 
context that may be important to understand. 



C_Mission 

Data that reference school or district mission, 
or lack thereof and explain what this mission 
is. 



C_Race relations 

Data that reference situations or conditions 
that exist at a school or within a district 
because of issues related to race. 

Gangs (if race-related), tensions between parents or 
students of different races, school violence, tension 
between teachers and students. 

For data about community 
relations that are not race related, 
use C_Community relations. Does 
not need to be double-coded with 
C_Community relations. 

C_Reform History 

Data that reference past efforts to improve 
the school. Also includes perceptions of 
usefulness and rationales for past efforts. 

Restructuring, multiple reforms at once, school 
improvement plans, state or district intervention. 
Constantly changing reform strategies. 


C_Safety 

Data that reference the safety of the school 
environment both within the school and in the 
immediate neighborhood in which the school 
is situated. Also includes whether safety issues 
have changed over time. 

Student and staff safety in the school (either safe 
environment or dangerous environment). Safety 
level of the school in the community or immediate 
neighborhood. 

For specific data related to actions 
or strategies designed to improve 
school safety, use the student 
supports code. 

C_Size 

Data that reference the size of classrooms, 
school or district and how that defines or 
impacts the case study school. 

Large class sizes, small school, large school, 
declining enrollment. 
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Code 

Definition 

Examples 

Exclusions 

C_Stability 

Data that reference the extent to which 
stability exists within the school or district. 
Stability covers leadership, coaches, external 
providers, and teachers. Includes whether the 
degree of stability has changed over time and 
why. 

Leadership churn, teacher turnover. 

Don't use for student mobility, use 
C_Student population. 

C Student population 

Data that reference the types of students that 
attend the school or the district. Also includes 
issues related to student mobility since the 
extent of mobility would directly impact the 
student population. 

Includes race, socioeconomic status, EL, special 
education, socio-emotional challenges faced by 
students. 
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TEACHER-SPECIFIC (T): These codes apply to segments of data that are associated with individual teacher participants. Primarily, they are used 
to capture teachers' goals for their students, as well as data on how teachers are gauging success in their classrooms. 


Code 

Definition 

Examples 

Exclusions 

T_Goals for students 

Data that reference teachers' descriptions of 
the goals they've established for their 
students. 


No need to use the teacher 
multipurpose code here. 

T_Measuring success 

Data that reference how teachers measure 
their students success in meeting the goals 
that they established for them. 


No need to use the teacher 
multipurpose code here. 

T_Teacher Quality 

Data that reference students/teachers/ 
parents' perceptions about what makes a 
high-quality teacher. This question comes up 
mostly in the student and teacher focus 
groups. 

Group work, patience, attention to all students, 
ability to explain concepts in multiple ways, 
encouraging, supportive, communicates well with 
parents. 

No need to use the teacher 
multipurpose code here. 
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Appendix B. Technical Approach to Qualitative 
Analyses 


Exhibit B.l. 

Perceived External Context of Core Sample Schools 


Summary 

This analysis examines perceptions of the external context for the core sample schools. See 
Chapter 3 for a discussion of the analysis, including analytic results. 

Technical Detail 

Data Sources 

• Interviews with district administrators, principals, teachers, and instructional coaches, 
including the following questions to elicit responses on the external context for the 
schools (Note that information may also have been obtained through other points in the 
interview, not just in direct response to the questions listed below.): 

District administrator 

o How would you briefly describe your district to someone who has never been here 
before, including its strengths, challenges, and priorities? 

Principal, teacher, and instructional coach 

o How would you briefly describe your school to someone who has never been here 
before, including its strengths and challenges? 

• Focus groups with teachers, students, and parents, including the following questions to 
elicit responses on the external context for the schools (Note that information may also 
have been obtained through other points in the focus group, not just in direct response 
to the questions listed below.): 

Teacher and parent 

o How would you describe this school to someone who has never been here before? 
What are the major strengths? Are there some things you would like to see that 
would make this school better? 

Student 

o How would you describe your school to a friend who goes to a different school? 

• Observations by site visit teams 

Stage 1: 
Qualitative 
Data Analysis 
Procedures 

Qualitative data were coded according to the codebook in Appendix A (see Chapter 2 for a 
description of the coding procedures). To examine respondent perceptions of the external 
context, site visitors responded to the following question in the online data repository based 
on all coded data associated with school context [C_XXj: 

• From the stakeholders' perspective, what are the defining features of the school context 
and history? These could pertain to the geographic setting, school facilities, student 
population, or special features of the school. Regarding the school's history, these could 
include changes in demographics, race relations, achievement levels, court decisions, 
reform history, etc. From which respondents did you draw these data? Please note the 
degree of consistency across stakeholders. 
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Exhibit B.l. 

Perceived External Context of Core Sample Schools (continued from previous page) 



Technical Detail 

Stage 1: 
Qualitative 
Data Analysis 
Procedures 
(continued from 
previous page) 

As part of the repository, site visitors were also asked to add their own observations of the 
external environments of their schools. After the first site visitor completed the data 
repository for his or her school(s), the second site visitor reviewed the responses to ensure 
the accuracy and completeness of the entries. 

Using responses to the question above on the defining features of the school context, analysts 
identified data that reflected the external context of the school (e.g., respondents' reports of 
crime, neighborhood safety). These responses included statements such as the following: 

• "[The school] is located in a high-poverty neighborhood that has a long history of 
violence. The school is surrounded by public housing projects, and the majority of 
students at the school live in these communities. Teachers, the principal, and parents all 
discussed the impact of [the neighborhood] on their students. Many see violent crime on 
a regular basis, are either involved with or are witness to abusive relationships with 
parents or relatives, and have family members that are incarcerated." 

• "[The school] is in a small town. The superintendent and external provider both 
emphasized the advantages of this: the friendly, safe, 'small town feel' and investment 
the community has in the school." 

Stage 2: 
School 
Classification 
Procedures 

Using the repository responses to the questions above, analysts categorized schools using the 
classification scheme on perceived external context described below. When the classifications 
were complete, the site lead for each school reviewed and verified the categorizations for his 
or her school(s). In cases of disagreement, the analysts and site lead returned to the coded 
data for the particular school(s) in question to resolve the disagreement. For this analysis, 
"respondents" refers to the preponderance of evidence from all respondent groups listed 
above. 

Traumatic context 

• Respondents reported prevalent crime, dilapidated buildings, unstable families, poor 
race relations (e.g., recent race-related riots), gang activity, high rates of home 
foreclosures, or high rates of unemployment; AND/OR 

• Site visitors observed dilapidated buildings, boarded-up windows on nearby homes, 
graffiti, police activity, broken windows on nearby homes and parked cars, litter 
(including beer cans and/or bottles on school grounds or in the neighborhood), or 
foreclosure signs in the nearby neighborhood. 

Depressed context 

• Respondents reported prevalent crime, unstable families, school or neighborhood 
disrepair, gang activity, high rates of home foreclosures, or high rates of unemployment, 
but also recent improvements in the neighborhood in which the school is situated (e.g., 
respondents reported a decrease in crimes in the area, repairs being made to 
neighborhood parks or school buildings, or new housing developments); AND/OR 

• Site visitors observed dilapidated buildings, boarded-up windows on nearby homes, 
graffiti, broken windows on nearby homes or parked cars, litter, or foreclosure signs in 
the neighborhood, but also new housing developments, construction or work crews 
working in nearby parks or buildings or the school building itself. 
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Exhibit B.l. 

Perceived External Context of Core Sample Schools (continued from previous page) 



Technical Detail 

Stage 2: 

School 
Classification 
Procedures 
(continued from 
previous page) 

Benign context 

• Respondents described the surrounding community as safe, stable (e.g., low rates of 
transiency or student mobility), close-knit (e.g., neighbors know and provide support to 
one another), or with higher levels of home ownership and building repair than 
neighboring communities; AND/OR 

• Site visitors observed well-maintained yards, very little or no litter in the surrounding 
area or on school grounds, relatively new school buildings, or construction or work crews 
working in nearby parks or buildings. 

Caveats 

These school-level classifications are not based on objective, quantitative indicators of the 
external context of the school. They are based on the reports of interview and focus group 
participants as well as observations of the physical environment and surrounding areas by the 
site visit team, which were informal and not systematic in nature. 

Notes 

Includes 25 core sample schools. 
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Exhibit B.2. 

Perceived Funding and Resource Constraints 


Summary 

This analysis examines perceptions of funding and resource constraints for the core sample 
schools. See Chapter 3 for a discussion of the analysis, including analytic results. 

Technical Detail 

Data Sources 

• Interviews with district administrators, principals, teachers, and instructional coaches, 
including the following questions to elicit responses on funding and resource constraints 
(Note that information may also have been obtained through other points in the 
interview, not just in direct response to the questions listed below.): 

District administrator and principal 

o Why do you believe this school has remained low-performing year after year? 

What has hindered improvement efforts in the past (prior to receiving SIG funds)? 

o Generally speaking, what do you think these persistently low-performing schools 
need to improve performance (e.g., funding, expertise, staff capacity, etc.)? 

Teacher and instructional coach 

o How would you briefly describe your school to someone who has never been here 
before, including its strengths and challenges? 

• Focus groups with teachers, including the following question to elicit responses on 
funding and resource constraints (Note that information may also have been obtained 
through other points in the focus group, not just in direct response to the question listed 
below.): 

o How would you describe this school to someone who has never been here before? 
What are the major strengths? Are there some things you would like to see that 
would make this school better? 

• Teacher survey data 

Stage 1: 
Qualitative 
Data Analysis 
Procedures 

To examine the extent to which perceived funding and resource constraints, prior to receiving 
School Improvement Grants (SIG), were a barrier preventing or constraining schools from 
making desired improvements, analysts reviewed all coded data associated with contextual 
information about funding levels [C_Level of funding], as well as the full principal interview 
transcripts. Qualitative data were coded according to the codebook in Appendix A (see 
Chapter 2 for a description of the coding procedures). 

Using these data, analysts identified data on the types of fiscal challenges reported by 
respondents and the kinds of improvement strategies that had been cut due to fiscal 
constraints, noting how many and which types of respondents provided data. These data 
included statements such as the following: 

• "[The district] has seen such dramatic funding cuts over the last three years that it has 
affected everything. We can't afford it. ..to think maybe new teachers are going to get 
their pink slip and have that worry; 1 see long-term ramifications— are you really going to 
get people that want to see a career in education?" (union representative) 

• "The school district is strapped for resources. They don't have money to provide books, 
clean the building. ..and teachers that are good will leave as soon as they can, trying to 
get to a magnet school or middle-class neighborhood school that has more resources...." 
(external provider) 

• "1 don't think we lack for internal staff to support the staff, to support the work." 
(principal) 
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Exhibit B.2. 

Perceived Funding and Resource Constraints (continued from previous page) 



Technical Detail 

Stage 2: 
Teacher Survey 
Data Analysis 
Procedures 

Analysts incorporated data from the school resources scale, which included four items in 
which teachers were asked how much of a challenge each of the following were to improving 
their school: large class-size and/or caseload, inadequate or substandard facilities, too few 
textbooks and other instructional materials, and textbooks and instructional materials that 
are not aligned with state standards. See Exhibit 2.7 for a detailed description of this scale. 

Stage 3: 
School 
Classification 
Procedures 

Using the coded data and teacher survey data, analysts categorized schools using the 
classification scheme on perceived funding and resource constraints described below. When 
the classifications were complete, the site lead for each school reviewed and verified the 
categorizations for his or her school(s). In cases of disagreement, the analysts and site lead 
returned to the coded data for the particular school(s) in question to resolve the 
disagreement. For this analysis, "respondents" refers to individuals from the respondent 
groups listed above. 

Perceived as a barrier to school improvement 

• Qualitative data: At least three respondents mentioned cuts to programs; or at least two 
respondents described cuts to programs and articulated how these had impacted or 
would impact student achievement; or at least two respondents mentioned cuts to 
programs and described "dire" circumstances (e.g., 30 percent of staff being laid off); 
AND 

• Survey data: School resources scale average was at least 0.5 standard deviations (0.40) 
below the scale mean (2.63). 

Perceived as a moderate challenge 

• Qualitative data: At least one respondent mentioned only minor cuts; AND/OR 

• Survey data: School resources scale average was within 0.5 standard deviations (0.40) of 
the scale mean (2.63); OR 

Perceptions were mixed 

• Qualitative data: Respondents disagreed (in interview data) about the adequacy of 
school resources; AND/OR 

• Qualitative data and teacher survey data did not match (e.g., school resources scale 
average was at least 0.5 standard deviations [0.40] below the scale mean [2.63], but 
qualitative data indicated "no fiscal constraints"). 

Not perceived as a challenge 

• Qualitative data: No respondent mentioned funding cuts, fiscal constraints, or resource 
constraints as a challenge for the school; AND 

• Survey data: School resources scale average was at least 0.5 standard deviations (0.40) 
above the scale mean (2.63). 

Caveats 

This analysis is not a systematic examination of fiscal constraint, but rather an aggregate 
reflection of the perceptions of respondents regarding resources at their schools. Actual 
school spending levels prior to receipt of SIG are included in the report (see Chapter 3) but 
are not considered in this analysis. 

For four schools, school classifications were based on interview and focus group data only, as 
response rates for the teacher survey did not meet the 50 percent threshold. 

Notes 

Includes 25 core sample schools. 


B-5 


Case Studies of Schools Receiving School Improvement Grants 


Exhibit B.3. 

Perceived Performance Problems 


Summary 

This analysis examines perceptions of the performance problems for the core sample schools. 
See Chapter 3 for a discussion of the analysis, including analytic results. 

Technical Detail 

Data Sources 

• Interviews with district administrators, principals, teachers, and instructional coaches, 
including the following questions to elicit responses on school performance problems 
(Note that information may also have been obtained through other points in the 
interview, not just in direct response to the questions listed below.): 

District administrator and principal 

o Why do you believe this school has remained low-performing year after year? 

What has hindered improvement efforts in the past (prior to receiving SIG funds)? 

o Generally speaking, what do you think these persistently low-performing schools 
need to improve performance (e.g., funding, expertise, staff capacity, etc.)? 

Teacher and instructional coach 

o How would you briefly describe your school to someone who has never been here 
before, including its strengths and challenges? 

• Focus groups with teachers, including the following question to elicit responses on 
school performance problems (Note that information may also have been obtained 
through other points in the focus group, not just in direct response to the question listed 
below.): 

o How would you describe this school to someone who has never been here before? 
What are the major strengths? Are there some things you would like to see that 
would make this school better? 

Stage 1: 
Qualitative 
Data Analysis 
Procedures 

Qualitative data were coded according to the codebook in Appendix A (see Chapter 2 for a 
description of the coding procedures). To examine respondent perceptions of performance 
problem(s), site visitors responded to the following question in the online data repository 
based on the coded data (specific codes used are provided in brackets): 

• How did school stakeholders describe the performance problem in their school? That is, 
to what did they attribute their school's history of low performance? Please document 
the different data sources that contribute to your response. This should not just be a 
summary list, but a description of the story that school stakeholders tell about persistent 
failure. Is there a common, shared perspective or different interpretations? [“"Problem 
definition, “"Challenge, C_behavior, C_reform history] 

After the first site visitor completed the data repository for his or her school(s), the second 
site visitor reviewed the responses to ensure the accuracy and completeness of the entries. 
Using the responses to the question above on respondents' perceptions of the reasons for 
their schools' history of low performance, analysts identified the following 11 domains of 
explanations associated with the performance problems: 
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Exhibit B.3. 

Perceived Performance Problems (continued from previous page) 



Technical Detail 

Stage 1: 
Qualitative 
Data Analysis 
Procedures 
(continued from 
previous page) 

Explanations related to students 

• Student behavior, including discipline problems, poor student attendance, behavior 
problems, a lack of student motivation, students with emotional issues, or weak student 
engagement. 

• Challenges with specific student subgroups, including specific challenges associated with 
students with severe mental health challenges (e.g., autism, severe emotional 
disturbance), a high concentration of English language learners, a large number of 
special education students, or a variety of languages spoken in students' homes (other 
than English). 

• Lack of student preparation in prior grades. 

Explanations related to teachers 

• Poor instruction/low teacher guality, including teachers lacking the capacity to provide 
effective instruction, a lack of instructional rigor, or a lack of consistent instructional 
practices or programs. 

• Teacher recruitment or retention (i.e., problems recruiting teachers and high teacher 
turnover). 

Explanations related to leadership 

• School leadership, including instability in administrator or school leaders, a lack of (or 
poor) school leadership, or poor communication skills of school leaders. 

• District leadership and support, including poor leadership by district officials, instability 
at the district level, a lack of district support (i.e., in terms of resources), disorganization 
within the central office, or a lack of communication from the school district. 

Explanations related to the school as a whole or the community 

• School external context, including the school's geographic location; the demographics of 
the students (e.g., concentration of one ethnic group, high poverty or migrant families); 
a transient or unstable community or neighborhood where families move in and out of 
the neighborhood, and few in the community have been there for more than a year; 
neighborhood or school safety; or problems related to violence in the neighborhood or 
the school. 

• School culture, including a culture of low expectations, a lack of consistency across 
school personnel's expectations for students, a divisive relationship between school 
leadership and teachers, a lack of a clear or shared vision for change, faculty and staff 
who are resistant to change, a lack of teacher collaboration, teacher isolation, poor staff 
morale, or overburdened teachers. 

• Parents/community, including limited parent engagement or involvement, a poor 
relationship with parents and the community, parents not being knowledgeable enough 
to help their children with schoolwork, parents or the broader community not 
demanding high academic achievement or rigor, or a lack of value placed on education 
by parents. 

• Inadeguate funding 
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Exhibit B.3. 

Perceived Performance Problems (continued from previous page) 



Technical Detail 

Stage 2: 

Classification 

Procedures 

Using the repository responses to the questions above, for each domain, analysts identified 
whether the explanation was perceived as a performance problem using the criteria described 
below. When the analysis was complete, the site lead for each school reviewed and verified 
the results for his or her school(s). In cases of disagreement, the analysts and site lead 
returned to the coded data for the particular school(s) in question to resolve the 
disagreement. 

Identified as perceived performance problem 

• At least one of the following respondent groups described challenges associated with 
the domain: at least two teachers, the principal, or a district administrator. 

• Note that other respondents, including instructional coaches, members of the school 
leadership team, parents, and students often volunteered corroborating responses, 
although their commentary alone did not suffice for the identification of a performance 
problem. 

Notes 

Includes 25 core sample schools. 
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Exhibit B.4. 

Perceptions of Locus of Responsibility for Performance Problems 


Summary 

This analysis examines whether respondents perceived the challenges facing their school as 
within their control (i.e., internal) or not (i.e., external). See Chapter 3 for a discussion of the 
analysis, including analytic results. 

Technical Detail 

Data Sources 

• Interviews with principals and teachers (No specific questions were asked. Information 
for this analysis may have been captured at multiple points in the interview.) 

• Focus groups with teachers (No specific questions were asked on the locus of 
responsibility for performance problems. Information for this analysis may have been 
captured at multiple points in the focus group.) 

• Teacher survey data 

Stage 1: 

Qualitative Data 

Analysis 

Procedures 

Qualitative data were coded according to the codebook in Appendix A (see Chapter 2 for a 
description of the coding procedures). To determine whether school respondents attributed 
their schools' performance problems to external versus internal factors, site visitors 
responded to the following question in the online data repository based on the coded data 
(specific codes used are provided in brackets): 

• How did school stakeholders describe the performance problem in their school? That is, 
to what did they attribute their school's history of low performance? Please document 
the different data sources that contribute to your response. This should not just be a 
summary list, but a description of the story that school stakeholders tell about 
persistent failure. Is there a common, shared perspective or different interpretations? 
[“"Problem definition, “"Challenge, C_behavior, C_reform history] 

After the first site visitor completed the data repository for his or her school(s), the second 
site visitor reviewed the responses to ensure the accuracy of the entries. 

Using responses to the question above on the schools' history of low performance, analysts 
identified data on respondent perceptions regarding responsibility for their school's 
performance problems. 

Stage 2: 
Teacher Survey 
Data Analysis 
Procedures 

Analysts incorporated data from a teacher survey item that measured "collective efficacy"— a 
Likert-scale item asking about level of agreement that "if teachers in this school work hard, 
we can meet our school's goals for student achievement." 

Because individual survey items are less precise than survey scales, schools were differentiated 
into just two groups with the survey data— above the mean and below the mean. 

Stage 3: 
School 
Classification 
Procedures 

Using the repository responses to the question above, analysts categorized schools using the 
classification scheme on responsibility for performance problems described below. When the 
classifications were complete, the site lead for each school reviewed and verified the 
categorizations for his or her school(s). In cases of disagreement, the analysts and site lead 
returned to the coded data for the particular school(s) in question to resolve the 
disagreement. For this analysis, "respondents" refers to at least two teachers or the principal. 


B-9 


Case Studies of Schools Receiving School Improvement Grants 


Exhibit B.4. 

Perceptions of Locus of Responsibility for Performance Problems 

(continued from previous page) 



Technical Detail 

Stage 3: 

School 
Classification 
Procedures 
(continued from 
previous page) 

Internal responsibility 

• Qualitative data: Respondents described their performance problems as being within 
the locus of control of the adults in the school; or respondents described the external 
context as challenging, but assumed responsibility for the school's history of low 
performance and did not describe these external challenges as insurmountable; or 
respondents described the external context in neutral terms, such as "This school is in a 
neighborhood with high crime"; AND 

• Survey data: School mean on the collective efficacy survey item was above the overall 
sample mean (3.07). (Two schools classified as having internal responsibility did not have 
adequate survey data, so their classification is based solely on the qualitative data.) 

Limited internal responsibility 

• Qualitative data: Respondents made statements that attributed the performance 
problems to the external context, but also described challenges within the locus of 
control of the adults in the school; or respondents attributed the school's history of low 
performance to a mix of internal and external factors; or respondents (i.e., at least two 
teachers or the principal) disagreed about the locus of responsibility; OR 

• Qualitative data and teacher survey data did not match (i.e., the collective efficacy item 
mean was below the overall sample mean [3.07], but interview and focus group data 
indicated "internal responsibility"; or the collective efficacy item mean was above the 
overall sample mean, but interview and focus group data indicated "external 
responsibility"). 

External responsibility 

• Qualitative data: Respondents explicitly identified factors external to the school as 
responsible for the performance problem and did not attribute the history of low 
performance to any factors internal to the school (e.g., school culture, instruction, 
leadership, collaboration); or respondents made statements that attributed the 
performance problems to others outside of the school, such as "Parent participation is 
why we are failing. That is the foundation"; AND 

• Survey data: School mean on the collective efficacy survey item was below the overall 
sample mean (3.07). (One school classified as having external responsibility did not have 
adequate survey data, so its classification is based solely on the qualitative data.) 

Caveats 

All core sample schools faced at least some external challenges, whether from the immediate 
context of the school (e.g., isolation or lack of security), lack of district support, limited 
financial resources, or high levels of student poverty and associated challenges. Therefore, it 
would not be surprising for respondents to describe these factors to the site visitors. The 
distinction for this analysis is whether respondents associated performance problems with 
the context. 

Notes 

Includes 25 core sample schools. 
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Exhibit B.5. 

Perceptions of Transformational Leadership 


Summary 

This analysis examines the extent to which principals at core schools were perceived as being 
characterized by or engaged in transformational leadership. See Chapter 4 for a discussion of the 
analysis. 


Technical Detail 

Data Sources 

• Interviews with teachers and instructional coaches, including the following questions to elicit 
responses on principal leadership (Note that information may also have been obtained through 
other points in the interview, not just in direct response to the questions listed below.): 

Teacher 

o How would you characterize the leadership of this school? Who are the key leaders and 
what do they do to move the school forward and support you as a teacher? To what 
extent do you think they are effective in leading particular aspects of the school? 
Instructional coach 

o As an instructional coach, to what extent are you satisfied with the support you get 
from the school leadership? Teachers? Please explain the ways in which support is 
evident or not evident. 

• Focus groups with school improvement teams and teachers, including the following questions 
to elicit responses on principal leadership (Note that information may also have been 
obtained through other points in the focus group, not just in direct response to the questions 
listed below.): 

School improvement team 

o How would you describe the leadership at this school? 

■ Who provides leadership? To what extent does the principal delegate 
leadership/management responsibilities to others? To what extent do you think 
the school and its leadership have the capacity to implement these improvement 
approaches and strategies? Does the principal have the appropriate leadership 
skills? 

Teacher 

o To what extent do you think the school's environment is conducive to teaching and 
learning? 

■ Does the school have high expectations for its students? Does your school have 
academic and behavioral standards? Do they promote or hinder success? 

• Teacher survey data 

Stage 1: 
Qualitative 
Data Analysis 
Procedures 

To examine perceptions of the principals' transformational leadership, analysts reviewed all coded 
data associated with school leadership [*School leader and $Leadership], as well as portions of 
transcripts related to the questions identified above. Qualitative data were coded according to the 
codebook in Appendix A (see Chapter 2 for a description of the coding procedures). 

Using these data, analysts identified 19 principal qualities, which were then compared against the 
21 qualities of effective leadership delineated by Waters et al. (2003) in their meta-analysis of 
principal leadership studies. This process yielded a set of 11 qualities that were common to both 
analyses, of which 8 were qualities characteristic of transformational leaders and 3 were qualities 
uncharacteristic of transformational leaders (although the labels may differ slightly, analysts 
determined that the content of the qualities emerging from the case study data was consistent 
with that in Waters et al. [2003]): 
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Perceptions of Transformational Leadership (continued from previous page) 



Technical Detail 

Stage 1: 

Qualitative 

Data Analysis 

Procedures 

(continued 

from previous 

page) 

Qualities characteristic of transformational leaders 

• Accessible/welcomes input. Easy to approach and communicate with, encourages teachers 
to contribute ideas. Typically described as having an "open door policy." 

• Supportive of staff. Takes into consideration the needs of the teachers and other 
administrators. Typically described as "understanding" and "helpful." 

• Visible/known to school community. Has frequent interactions with staff, students, parents, 
and community. Typically described as a "visible leader." 

• Visionary. Acts as an agent of change and expresses a clear direction for where the school is 
headed. 

• Enthusiastic. Inspires staff and students and takes on a positive attitude. 

• Communicative. Establishes clear communication systems for teachers, staff, and 
parents/community members. 

• Develops leaders. Gives opportunities to teachers and staff to take become leaders and 
increase their capacity as leaders. 

• High expectations. Establishes high expectations for staff and students. Typically described 
as "believing in kids and staff." 

Qualities uncharacteristic of transformational leaders 

• Poor communication. Does not establish clear communication systems for teachers, staff, 
and parents/community members. 

• Unsupportive ofstaff/bad rapport. Does not take into consideration the needs of teachers 
and staff. 

• Authoritarian. Does not ask for teacher and staff input, and makes decisions individually. 
Typically described as not letting teachers "have a voice." 

Stage 2: 
Teacher 
Survey Data 
Analysis 
Procedures 

Analysts incorporated data from the principal trust scale, which included six items in which 
teachers were asked about their principal. See Exhibit 2.7 for a detailed description of this scale. 

Stage 3: 
School 
Classification 
Procedures 

Using the coded interview and focus group data and the teacher survey data, analysts 
categorized principals using the classification scheme on transformational leadership described 
below. When the classifications were complete, the site lead for each school reviewed and 
verified the categorizations for his or her school(s). In cases of disagreement, the analysts and 
site lead returned to the coded data for the particular school(s) in question to resolve the 
disagreement. For this analysis, "respondents" refers to all respondent groups listed above. 

High on continuum 

• Qualitative data: The principal demonstrated at least three of the eight qualities related to 
transformational leadership (demonstrated qualities refer to those mentioned by at least 
two respondents); AND 

• Survey data: Principal trust scale average was at least 0.5 standard deviations (0.35) above 
the scale mean (3.09). 

Middle of continuum 

• The principal did not demonstrate evidence to be categorized in the high or low end of the 
continuum. 

Low on continuum 

• Qualitative data: The principal demonstrated none of the eight qualities related to 
transformational leadership; AND 

• Survey data: Principal trust scale average was at least 0.5 standard deviations (0.35) below 
the scale mean (3.09). 
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Exhibit B.5. 

Perceptions of Transformational Leadership (continued from previous page) 



Technical Detail 

Caveats 

Classifications are based on school staff reports during the spring 2011 site visits and teacher 
survey. The analysts were not able to make direct evaluations of the strength of principals' 
transformational leadership. 

Notes 

Includes 23 of 25 core sample schools. Two schools were excluded from this analysis due to 
insufficient qualitative data (i.e., respondents did not explicitly address the topic). 
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Exhibit B.6. 

Perceptions of Instructional Leadership 


Summary 

This analysis examines the extent to which principals at core sample schools were perceived as 
being characterized by or engaged in instructional leadership. See Chapter 4 for a discussion of 
the analysis. 


Technical Detail 

Data Sources 

• Interviews with teachers and instructional coaches, including the following questions to 
elicit responses on principal leadership (Note that information may also have been 
obtained through other points in the interview, not just in direct response to the 
questions listed below.): 

Teacher 

o How would you characterize the leadership of this school? Who are the key leaders 
and what do they do to move the school forward and support you as a teacher? To 
what extent do you think they are effective in leading particular aspects of the school? 
Instructional coach 

o As an instructional coach, to what extent are you satisfied with the support you get 
from the school leadership? Teachers? Please explain the ways in which support is 
evident or not evident. 

• Focus groups with school improvement teams and teachers, including the following 
questions to elicit responses on principal leadership (Note that information may also have 
been obtained through other points in the focus group, not just in direct response to the 
questions listed below.): 

School improvement team 

o How would you describe the leadership at this school? 

■ Who provides leadership? To what extent does the principal delegate 
leadership/management responsibilities to others? To what extent do you think 
the school and its leadership have the capacity to implement these 
improvement approaches and strategies? Does the principal have the 
appropriate leadership skills? 

Teacher 

o To what extent do you think the school's environment is conducive to teaching and 
learning? 

■ Does the school have high expectations for its students? Does your school have 
academic and behavioral standards? Do they promote or hinder success? 

• Teacher survey data 

Stage 1: 
Qualitative 
Data Analysis 
Procedures 

To examine perceptions the principals' instructional leadership, analysts reviewed all coded 
data associated with school leadership [*School leader and $Leadership], as well as portions of 
transcripts related to the questions identified above. Qualitative data were coded according to 
the codebook in Appendix A (see Chapter 2 for a description of the coding procedures). 

Using these data, analysts identified data on activities associated with instructional leadership 
of the principal (e.g., principals who focused on academics and the academic achievement of 
students, and who were in the classroom providing feedback on instruction and/or curriculum), 
noting how many and which types of respondents provided data. Examples of quotations 
defining the instructional leadership quality include the following: 

• "We have a principal that knows what she is doing, that has experience. She doesn't say 
that you need to do this without knowing what you're actually supposed to be doing. 
Whatever we're working on, she is involved. ..She attends all of our team meetings, she is 
in our classrooms, and she calls herself our instructional leader. 1 like how specific she is 
when she comes into our classrooms." ( teacher ) 

• "The current principal is the best. Our school is moving towards academics." (teacher) 
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Exhibit B.6. 

Perceptions of Instructional Leadership (continued from previous page) 



Technical Detail 

Stage 1: 
Qualitative 
Data Analysis 
Procedures 
(continued from 
previous page) 

• "The principal had an opportunity to go into our school, and he saw the difficulties. He 
really picked apart our curriculum and asked us: 'What is working? What is making 
teaching here work for you? What is making your kids be successful? What is not working 
so that we can get something on board to fix that?'" (teacher) 

Stage 2: 
Teacher 
Survey Data 
Analysis 
Procedures 

Analysts incorporated data from the instructional leadership scale, which included seven items 
in which teachers were asked about various attributes of their principal's instructional 
leadership. See Exhibit 2.7 for a detailed description of this scale. 

Stage 3: 
School 
Classification 
Procedures 

Using the coded interview and focus group data and the teacher survey, analysts categorized 
principals using the classification scheme on instructional leadership described below. When 
the classifications were complete, the site lead for each school reviewed and verified the 
categorizations for his or her school(s). In cases of disagreement, the analysts and site lead 
returned to the coded data for the particular school(s) in question to resolve the 
disagreement. For this analysis, "respondents" refers to individuals from respondent groups 
listed above. 

The threshold for classifications procedures using qualitative data may be perceived as low, 
but in nearly all cases the threshold was exceeded. Moreover, the use of survey data supports 
the validity of these classifications. 

High on continuum 

• Qualitative data: At least two school-level respondents described principal activities 
associated with instructional leadership; AND 

• Survey data: Instructional leadership scale average was at least 0.5 standard deviations 
(0.33) above the scale mean (3.12). 

Middle of continuum 

• The principal did not demonstrate evidence to be categorized in the high or low end of 
the continuum. 

Low on continuum 

• Qualitative data: Less than two school-level respondents mentioned principal activities 
associated with instructional leadership; AND 

• Survey data: Instructional leadership scale average was at least 0.5 standard deviations 
(0.33) below the scale mean (3.12). 

Caveats 

Classifications are based on school staff reports during the spring 2011 site visits and teacher 
survey. The analysts were not able to make direct evaluations of principals' strength as 
instructional leaders. 

Notes 

Includes 21 of 25 core sample schools. Four schools were excluded from this analysis due to 
not meeting the 50 percent response rate threshold on the teacher survey. 
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Exhibit B.7. 

Perceptions of Strategic Leadership: Theories of Action as Reported by Principals 


Summary 

This analysis examines the extent to which principals at core sample schools demonstrated a 
theory of action. See Chapter 4 for a discussion of the analysis. 

Technical Detail 

Data Sources 

• Interviews with principals, including the following questions to elicit responses on the 
theory of action (Note that information regarding the theory of action may also have 
been obtained through other points in the interview, not just in direct response to the 
questions listed below. Data on the rationale for selecting particular improvement 
actions, intended outcomes of the improvement actions, and underlying assumptions 
were generally prompted from the questions below): 

Defining the performance problem 

o Why do you believe this school has remained low performing year after year? What 
has hindered improvement efforts in the past (prior to receiving SIG funds)? 
o How would you describe your school to someone who has never been here before, 
including its strengths and challenges? 

Identifying improvement actions to address the performance problem 
o Generally speaking, what do you think your school needs to improve student 
performance (e.g., funding, expertise, staff capacity)? 
o What is your main priority as principal? 

o Could you describe the specific improvement strategies your school has 
implemented this school year? How have you prioritized these strategies? 

Stage 1: 
Defining the 
Concept 

As noted in Chapter 4, a theory of action, broadly stated, is the implicit or explicit set of 
operational assumptions regarding how the change process will unfold in a given school. To 
operationalize a principal's "theory of action," the study team identified five elements of a 
theory of action: (1) defining the performance problem; (2) identifying a set of improvement 
actions or primary levers of change to address the performance problem; (3) providing a 
rationale for selecting those improvement actions; (4) identifying the intended outcomes of 
those strategies; and (5) explaining the explicit and interrelated assumptions underlying the 
change process in a school. 

Stage 2: 

Qualitative Data 

Analysis 

Procedures 

Using this five-element framework, analysts coded the full principal interview transcripts to 
capture whether each of these five elements of a theory of action were present in the 
principal's interview. This process enabled analysts to examine the degree to which principal 
responses to certain key questions tied together, built on one another, and generally aligned 
with what the principal stated throughout the course of the interview, and thereby, allowing 
the analysts to obtain a comprehensive understanding of if and how principals articulated the 
five components of their theories of action. 

Using the coded principal interviews, analysts summarized the data to produce a narrative for 
each principal's theory of action. When synthesizing the elements of a principal's theory of 
action, they considered all five elements of the theories of action — performance problem, 
actions and primary levers of change, intended outcomes, rationale, and assumptions. This 
narrative included statements such as the following: 

• "The principal mentions that he would not consider the school as 'persistently low 
achieving'— he points to the accomplishments of the school (increased number of 
students enrolled in 2- and 4-year colleges and increased number of students receiving 
scholarships), and argues that the school was placed on the list of 'persistently low 
achieving' because of their performance on the state assessments (which do not take into 
consideration the fact that the school has a large population of newcomer ELL students). 
The principal mentions a long list of strategies— some of which very clearly tie into his 
goals and some which don't; but, he argues that at the center of everything is 'student 
success.'" 
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Exhibit B.7. 

Perceptions of Strategic Leadership: Theories of Action as Reported by Principals 

(continued from previous page) 



Technical Detail 

Stage 2: 

Qualitative Data 
Analysis 
Procedures 
(continued from 
previous page) 

• "The principal mentioned that the school has high rates of violence and safety issues. 

When the principal first arrived at the school, he said he first focused on repairing the 
'cosmetic stuff.' However, he emphasized the fact that the culture of low expectations was 
the main contributing issue to low performance at the school; he stated that it's one thing 
he has the power to change (since he has no control over community issues of violence). 
He also expressed the assumption that "phenomenal teachers" will lead to school 
improvement. This principal has a clear idea of what the school needs to improve and 
what steps he needs to take to get there. He mentions five 'concrete standards' to which 
PD is aligned and he mentions that his coaches provide teachers with feedback on this; 
additionally, he states how a system of coaches, professional development, monitoring, 
and school leadership will lead to improved teacher quality and school culture. The 
majority of the mentioned strategies reinforces one another and fit into a coherent plan 
for school improvement." 

Stage 3: 
School 
Classification 
Procedures 

For each element of the theory of action, analysts first determined whether the principal 
exhibited the element by examining the extent to which the principal could clearly articulate the 
element. Summary narratives were then used to assess the extent to which the elements 
mutually reinforced each other, or the coherence of the principal's theory of action. Analysts 
then classified the core sample schools based on the classification scheme on theories of action 
described below. When the classifications were complete, the site lead for each school 
reviewed and verified the categorization for his or her school(s). In cases of disagreement, the 
analysts and site lead returned to the coded data for the particular school(s) in question to 
resolve the disagreement. 

High on continuum 

• The principal exhibited at least four of the five elements of a theory of action; AND 

• The principal demonstrated that these elements mutually reinforce each other. 

Mid-High on continuum 

• The principal exhibited at least three of the five elements of a theory of action; AND 

• The principal expressed that some of these elements mutually reinforced one another, but 
not all of the elements reinforced one another or related to the perceived performance 
problem. 

Mid-Low on continuum 

• The principal exhibited less than three of the five elements of a theory of action; AND 

• The principal expressed that most of these elements were externally driven and most of 
the improvement strategies did not mutually reinforce one another or address the 
perceived performance problem. 

Low on continuum 

• The principal exhibited less than three of the five elements of a theory of action; AND 

• The principal was not able to or did not articulate certain components of the plan. 


B-17 


Case Studies of Schools Receiving School Improvement Grants 


Exhibit B.7. 

Perceptions of Strategic Leadership: Theories of Action as Reported by Principals 

(continued from previous page) 



Technical Detail 

Caveats 

This analysis focuses on principals only and is not a systematic analysis of the theories of action 
held by all respondents. Thus, the analysis is not intended to represent the theory of action at 
the school level. Instead, the analysis is meant to provide insight into the theories of action 
articulated by the principals of the 25 core sample schools. Using principal interviews as the 
primary source for this analysis, however, presents the following limitations: 

• Although principals are typically perceived as the leaders of a school, there may be other 
systems of leadership within a school and respondents outside a school that hold their 
own theories of action (which may or may not align with the theory of action set forth by 
the principal). 

Principal interviews may not fully represent a principal's theory of action. Theories of action 
change over time, so it is likely that the theories of action articulated by the principals in spring 
2011 have since shifted. 

Moreover, because the study team did not ask the principals to explicitly state their theories of 
action, the data are inferred by answers to relevant interview questions. Additional data 
limitations may have affected the results for three schools: Two principal interviews were 
conducted in conjunction with a vice principal or other administrator at the school, and one 
principal interview transcript was not fully transcribed due to poor audio quality. 

Notes 

Includes 25 core sample schools. 
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Exhibit B.8. 

Improvement Actions Implemented by Core Sample Schools 


Summary 

This analysis examines improvement actions implemented by the core sample schools. See 
Chapter 5 for a discussion of the analysis, including analytic results. 

Technical Detail 

Data Sources 

• Interviews with district administrators, principals, teachers, instructional coaches, and 
external support providers, including the following questions to elicit responses on 
improvement actions (Note that information may also have been obtained through 
other points in the interview, not just in direct response to the questions listed below.): 

District administrator 

o For each school, what instructional improvement strategies were 

planned/implemented this school year? What is the rationale behind these 
strategies? 

o How are the SIG funds being used to support or stimulate specific improvement 
strategies? 

Principal 

o Could you describe the specific improvement strategies your school has 
implemented this school year? How have you prioritized these strategies? 

o How have you used the SIG funds in your school this year? 

Teacher and instructional coach 

o Are there some broad approaches or strategies that the school as a whole is 
following to reach its improvement goals? What are they and do you think they 
are appropriate or likely to be effective? 

• Focus groups with school improvement teams, teachers, and parents, including the 
following questions to elicit responses on improvement actions (Note that information 
may also have been obtained through other points in the focus group, not just in direct 
response to the questions listed below.): 

School improvement team 

o How do you think the SIG improvement approaches/strategies identified for the 
school will address the issues facing the school? Please describe how you think 
specific improvement strategies are working, or will work, to improve the school. 

Teacher 

o Can you describe some of the specific improvement strategies that were, or will 
be, adopted at your school this school year? 

Parent 

o What kinds of activities to improve things for students at this school have you 
observed or heard about? 

Stage 1: 
Identifying 
Improvement 
Actions 

Based on federal SIG guidance, the study's research questions, and prior research on school 
improvement, the study team identified the following strategies and actions, which were 
categorized into three broad categories or domains— improving the capacity of teachers and 
leaders and managing their performance (human capital management); improving the 
technical core of instruction (what students learn and how), and improving school conditions 
that support teaching and learning (see Chapter 1 for a discussion of the study's conceptual 
framework and the strategies and actions in key improvement domains): 


B-19 


Case Studies of Schools Receiving School Improvement Grants 


Exhibit B.8. 

Improvement Actions Implemented by Core Sample Schools (continued from previous page) 



Technical Detail 

Stage 1: 
Identifying 
Improvement 
Actions 
(continued from 
previous page) 

• Human capital management 

o Teacher replacement, specifically replacement of at least 50 percent of teachers. 
o Professional learning, including increase in "formal" professional development 
(workshops, seminars, etc.) and increase in professional development embedded 
in the school activities (teacher collaboration and planning, mentoring, etc.). 
o Evaluation and performance management, including revisions of the teacher 
evaluation system and use of incentives to recruit or retain teachers. 

• Technical core of instruction 

o Curriculum and instruction, such as changes to the curriculum or instruction in 
mathematics, ELA, or science; changes to curriculum or instruction for English 
Learners or special education students; hiring an instructional coach or content 
coach; RTI (Response to intervention); class size reduction; and use of technology. 
o Data use, or strategies to monitor student learning and frequent and transparent 
use of student outcome data to guide instructional decisions, 
o Learning time, including extended day, restructured day (e.g., block scheduling), 
weekend school, after-school, or before-school tutoring, and summer school. 
o Student supports, such as adding licensed support professionals, improving school 
safety, conducting home visits, offering medical services, providing college 
counselors/advisors, and providing extracurricular activities. 

• Conditions that support teaching and learning 

o School climate, such as policies/strategies to reduce tardiness/attendance, 
policies/strategies to reduce bullying, mandatory school uniforms, efforts to 
increase respect for diversity, and other rules/expectations for student behavior, 
o Parent and community supports, including hiring a parent liaison, developing new 
parent leaders, creating a parent center on campus, and running adult 
education/parenting classes. 

Stage 2: 
Qualitative 
Data Analysis 
Procedures 

Qualitative data were coded according to the codebook in Appendix A (see Chapter 2 for a 
description of the coding procedures). To examine implementation of the improvement 
actions described above, site visitors responded to several questions in the online data 
repository for each particular improvement action based on all coded data associated with 
improvement strategies [*Change Strategy]. Specifically, respondents were asked to identify 
respondent groups that identified a particular action, as well as to provide additional details 
on the implementation process, whether the action preceded SIG funding, the reason why 
specific improvement actions were implemented, and how divergent these processes were 
from the previous academic year. Questions for actions related to curriculum and instruction 
are provided below as an example of the repository questions on improvement actions: 
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Exhibit B.8. 

Improvement Actions Implemented by Core Sample Schools (continued from previous page) 


Stage 2: 
Qualitative 
Data Analysis 
Procedures 
(continued from 
previous page) 


Technical Detail 

• For each change strategy focused on curriculum or instruction listed below, please 
identify whether the following respondents identified this strategy as being 
implemented during the 2010-11 academic year (check as many as apply). [Curriculum; 
Instruction; Technology; Change strategy] 

External 

District Instructional Support 

Administrator Principal Teachers Coaches Provider 

Changes to math curriculum 
or instruction 

Changes to ELA curriculum or 
instruction 
Changes to science 
curriculum or instruction 
Changes to English Learners 
curriculum or instruction 
Changes to instruction for 
special education students 
Hiring an instructional or 
content coach 
RTI (Response to 
Intervention) 

Class size reduction 
Use of technology 
Other (please specify) 

Note: For respondent groups with multiple respondents, at least one respondent must have 
reported a particular strategy as being implemented. 


• In general terms, please rate the extent to which the curriculum and instruction 


practices in 2010-11 were divergent from practices in prior years? [Curriculum; 
Instruction; Technology; Change strategy] 
o Unable to answer/inadequate data 

o Nothing— or very little— has changed in the school's approach to this domain of 
activities 


o The change strategies included in this domain show some evidence of new 
strategies, but the fundamental approach or structure remain the same; changes 
could be characterized as "marginal" or "tweaks." 
o School staff report efforts to break notably with prior practice and provide specific 
examples that contrast prior to current practice. However, some practices within 
the domain appear untouched by change strategies (they were either reported as 
untouched, or there were insufficient data to determine if all practices were 
fundamentally changed). 

o According to reports from many stakeholders, all activities in this domain are new 
this school year or have been overhauled. No (or very few) aspects of the former 
approach in this domain have persisted. 

• Please list data sources for your rating in the question above. Were there multiple 
perspectives, or did respondents tend to agree? 
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Exhibit B.8. 

Improvement Actions Implemented by Core Sample Schools (continued from previous page) 



Technical Detail 

Stage 3: 

Categorization 

Procedures 

Using the repository responses to the questions above, for each action, analysts identified 
whether the improvement action was being implemented during the 2010-11 academic year 
using the criteria described below. When the analysis was complete, the site lead for each 
school reviewed and verified the results for his or her school(s). In cases of disagreement, the 
analysts and site lead returned to the coded data for the particular school(s) in question to 
resolve the disagreement. For this analysis, "respondents" refers to individuals from the 
respondent groups listed above. 

Identified as being implemented 

• At least three respondent groups identified the improvement action as being 

implemented. For certain improvement actions, two respondent groups were used as 
the threshold if analysts considered the respondents reporting the action to be key 
informants for the particular action (e.g., teachers and a coach reporting on coaching) or 
if analysts deemed it unreasonable for other school or district respondents to have deep 
knowledge about the action in question. 

Notes 

Includes 25 core sample schools. 
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Exhibit B.9. 

Impetus for Teacher Replacement 


Summary 

This analysis examines the impetus for teacher replacement for the core sample schools that 
replaced at least half of their teaching staff in 2010-11 (as identified through the analysis of 
implemented improvement actions discussed in Exhibit B.8). See Chapter 5 for a discussion of 
the analysis, including analytic results. 


Technical Detail 

Data Sources 

• Interviews with district administrators, principals, and teachers, including the following 
questions to elicit responses on the teacher replacement process (Note that information 
may also have been obtained through other points in the interview, not only in direct 
response to the questions listed below.): 

District administrator 

o For each school, what instructional improvement strategies were 

planned/implemented this school year? What is the rationale behind these 
strategies? 

o In SIG schools in which the principal and many staff members were replaced, how 
were decisions made about which staff to keep and which to let go? What was the 
rationale? 

Principal 

o Could you describe the specific improvement strategies your school has 
implemented this school year? How have you prioritized these strategies? 

o Were any teachers or other staff replaced? If so, when and which staff were 
replaced, and why? 

Teacher 

o Are there some broad approaches or strategies that the school as a whole is 

following to reach its improvement goals? What are they and do you think they are 
appropriate or likely to be effective? 

o Can you describe some of the specific improvement strategies that you know are, or 
will be, adopted at your school this school year as part of SIG? 

• Focus groups with teachers, including the following question to elicit responses on the 
teacher replacement process (Note that information may also have been obtained 
through other points in the focus group, not just in direct response to the question listed 
below.): 

o Can you describe some of the specific improvement strategies that were, or will be, 
adopted at your school this school year? 

Stage 1: 
Qualitative 
Data Analysis 
Procedures 

Qualitative data were coded according to the codebook in Appendix A (see Chapter 2 for a 
description of the coding procedures). To examine the impetus for teacher replacement, site 
visitors responded to the following questions based on the coded data (specific codes used are 
provided in brackets): 

• Did respondents indicate that the schools removed staff members who were unskilled 
and/or unmotivated? Please provide evidence. [$Staffing, *Change Strategy] 

• For turnaround schools, was there an additional, non-SIG reform model being 
implemented at the school, which also required replacement of 50 percent of staff, as 
evidenced by the principal and/or district administrators? If yes, please explain. 

[$Staffing, *Change Strategy] 

After the first site visitor completed the data repository for his or her school(s), the second site 
visitor reviewed the responses to ensure the accuracy of the entries. 
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Exhibit B.9. 

Impetus for Teacher Replacement (continued from previous page) 



Technical Detail 

Stage 2: 

Classification 

Procedures 

Using the responses to the questions above, analysts identified the following impetuses for 
teacher replacement: 

• Need to remove staff perceived as less skilled or motivated 

• Mandate ofSIG Turnaround model 

• Mandate ofnon-SIG improvement initiative that required the replacement of at least 
50 percent of the staff 

• Voluntary resignation. As part of a CMO takeover, teachers were required to reapply 
for their positions, but none of the teachers elected to stay. 

Identification criteria are described below. Data from teachers provided details about 
teacher replacement. When the analysis was complete, the site lead for each school 
reviewed and verified the results for his or her school(s). In cases of disagreement, the 
analysts and site lead returned to the coded data for the particular school(s) in question to 
resolve the disagreement. 

Identified as an impetus 

• A district administrator and/or the principal reported this impetus as the rationale for 
teacher replacement. 

Notes 

Includes 9 core sample schools that replaced at least half of their teaching staff in 2010-11. 
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Exhibit B.10. 

Teacher Replacement Process in Core Sample Schools 


Summary 

This analysis examines the teacher replacement process for the core sample schools that 
replaced at least half of their teaching staff in 2010-11 (as identified through the analysis of 
implemented improvement actions discussed in Exhibit B.8). See Chapter 5 for a discussion of 
the analysis, including analytic results. 


Technical Detail 

Data Sources 

Interviews with district administrators, principals, and teachers, including the following 
questions to elicit responses on the teacher replacement process (Note that information may 
also have been obtained through other points in the interview, not just in direct response to 
the questions listed below.): 

District administrator 

o For each school, what instructional improvement strategies were planned/ 
implemented this school year? What is the rationale behind these strategies? 

o How are the SIG funds being used to support or stimulate specific improvement 
strategies? 

o In SIG schools in which the principal and many staff members were replaced, how 
were decisions made about which staff to keep and which to let go? What was the 
rationale? 

Principal 

o Could you describe the specific improvement strategies your school has 
implemented this school year? How have you prioritized these strategies? 

o How have you used the SIG funds in your school this year? 

o Were any teachers or other staff replaced? If so, when and which staff were 
replaced, and why? 

Teacher 

o Are there some broad approaches or strategies that the school as a whole is 

following to reach its improvement goals? What are they and do you think they are 
appropriate or likely to be effective? 

o Can you describe some of the specific improvement strategies that you know are, or 
will be, adopted at your school this school year as part of SIG? 

• Focus groups with teachers, including the following question to elicit responses on 

the teacher replacement process (Note that information may also have been 
obtained through other points in the focus group, not just in direct response to the 
question listed below.): 

o Can you describe some of the specific improvement strategies that were, or will be, 
adopted at your school this school year? 

Stage 1: 
Qualitative 
Data Analysis 
Procedures 

Qualitative data were coded according to the codebook in Appendix A (see Chapter 2 for a 
description of the coding procedures). To examine the teacher replacement process, site 
visitors responded to the following questions in the online data repository based on the coded 
data (specific codes used are provided in brackets): 

• Provide any pertinent details related to the human resource strategies being 
implemented. For example, if the school had new teachers, how did the district recruit 
teachers and where did the previous teachers go? [$Staffing, *Change Strategy] 

• Provide details about the way in which teachers were identified to leave the school. In 
particular, if the least capable teachers left the school, who decided this, and how? 

Which respondents provided information on this topic? [$Staffing] 

• Did the principal have discretion over hiring decisions? If not, please explain. [$Staffing] 
After the first site visitor completed the data repository for his or her school(s), the second site 
visitor reviewed the responses to ensure the accuracy of the entries. 
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Exhibit B.10. 

Teacher Replacement Process in Core Sample Schools (continued from previous page) 



Technical Detail 

Stage 2: 

Classification 

Procedures 

Using the responses to the questions above and all coded data on staffing decisions and 
conditions [$Staffing], analysts identified two areas of focus related to the teacher 
replacement process: 

• Level of principal and teacher involvement in the replacement process 

• Limits on the pool of available teacher candidates caused by policy and contextual 
constraints 

Classifications are based on reports from the district administrator and/or the principal. Data 
from teachers provided additional details about teacher replacement. When the analysis was 
complete, the site lead for each school reviewed and verified the results for his or her 
school(s). In cases of disagreement, the analysts and site lead returned to the coded data for 
the particular school(s) in question to resolve the disagreement. 

The threshold for classifications procedures using qualitative data may be perceived as low; 
however, the principal and the district administrator were in the best position to provide data 
on these topics and their perspective should be required for classification. As noted, data from 
teachers also were taken into consideration. 

Notes 

Includes 9 core sample schools that replaced at least half of their teaching staff in 2010-11. 
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Exhibit B.ll. 

Perceptions of the Teacher Replacement Process 


Summary 

This analysis examines perceptions of the teacher replacement process (i.e., the rules governing 
the hiring and removal of staff and the extent to which principals and teachers were involved in 
the replacement process) for the core sample schools that replaced at least half of their teaching 
staff in 2010-11 (as identified through the analysis of implemented improvement actions 
discussed in Exhibit B.8). See Chapter 5 for a discussion of the analysis, including analytic results. 


Technical Detail 

Data Sources 

• Interviews with district administrators, principals, and teachers, including the following 
questions to elicit responses on the teacher replacement process (Note that information 
may also have been obtained through other points in the interview, not just in direct 
response to the questions listed below.): 

District administrator 

o For each school, what instructional improvement strategies were 

planned/implemented this school year? What is the rationale behind these 
strategies? 

o In SIG schools in which the principal and many staff members were replaced, how 
were decisions made about which staff to keep and which to let go? What was the 
rationale? 

Principal 

o Could you describe the specific improvement strategies your school has 
implemented this school year? How have you prioritized these strategies? 

o Were any teachers or other staff replaced? If so, when and which staff were 
replaced, and why? 

Teacher 

o Are there some broad approaches or strategies that the school as a whole is 

following to reach its improvement goals? What are they and do you think they are 
appropriate or likely to be effective? 

o Can you describe some of the specific improvement strategies that you know are, or 
will be, adopted at your school this school year as part of SIG? 

• Focus groups with teachers, including the following questions to elicit responses on the 
teacher replacement process (Note that information may also have been obtained 
through other points in the focus group, not just in direct response to the questions listed 
below.): 

o Can you describe some of the specific improvement strategies that were, or will be, 
adopted at your school this school year? 

o Do you think the improvement strategies fit the needs of the school and/or students? 

o What will be the greatest challenges to implementing these strategies? 

Stage 1: 
Qualitative 
Data Analysis 
Procedures 

Qualitative data were coded according to the codebook in Appendix A (see Chapter 2 for a 
description of the coding procedures). To examine respondent perceptions of the teacher 
replacement process, site visitors responded to the following questions in the online data 
repository based on the coded data (specific codes used are provided in brackets): 

• How did the addition of new teachers shape the culture and climate the core sample 
school? [$Staffing, *Change Strategy, *Capacity] 

o Overall, the addition of new teachers was perceived to be a positive change for the 
school, bringing new energy, improved morale, or ideas, 
o Overall, the turnover among teachers was perceived as disruptive (and perhaps 
demoralizing), creating more problems for the staff, 
o Other (please specify) 

• Please add details about your response above, if you would like. 
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Exhibit B.ll. 

Perceptions of the Teacher Replacement Process (continued from previous page) 



Technical Detail 

Stage 1: 
Qualitative 
Data Analysis 
Procedures 
(continued from 
previous page) 

After the first site visitor completed the data repository for his or her school(s), the second site 
visitor reviewed the responses to ensure the accuracy of the entries. 

Using responses to the questions above, analysts identified data on respondent perceptions 
regarding the teacher replacement process. These responses included statements such as the 
following: 

• The teacher in one focus group expressed mild concern that the change was disruptive, 
but, overall, the addition of new teachers was perceived to be a very positive influence in 
the school. The new teachers raised expectations, brought expertise, and were generally 
regarded as strong, collaborative teachers. One teacher said, "They are rising to the 
challenge, and they have high expectations," and "To me, it's like a rebirth." 

Stage 2: 
School 
Classification 
Procedures 

Using the repository responses to the questions above, analysts categorized schools using the 
classification scheme on perceptions of the teacher replacement process described below. 
When the classifications were complete, the site lead for each school reviewed and verified 
the categorizations for his or her school(s). In cases of disagreement, the analysts and site lead 
returned to the coded data for the particular school(s) in question to resolve the 
disagreement. For this analysis, "respondents" refers to all the respondent groups listed 
above. 

Positive 

• At least two respondents, regardless of respondent group, indicated that the new 
teachers introduced as part of the replacement process were beneficial (i.e., "bringing 
new energy," improving staff morale, increasing teacher quality); AND 

• No respondent described the teacher replacement in terms such as biased or unfair. 
Neutral 

• Respondents did not comment on the quality of the teacher replacement process or the 
quality of new teachers, or described the process in neutral terms, such as "another 
instance of change." 

Negative 

• At least two respondents, regardless of respondent group, indicated that the new 
teachers introduced as part of the replacement process were detrimental to the school 
(i.e., weakening staff morale, decreasing teacher quality); OR 

• At least two respondents, regardless of respondent group, described the teacher 
replacement process in terms such as biased or unfair. 

Caveats 

This analysis is not an examination of teacher effectiveness, and it is not intended to imply a 
causal connection between teacher replacement and changes in teacher effectiveness. Rather, 
this analysis is an aggregate reflection of the perceptions of respondents regarding the teacher 
replacement process. 

Notes 

Includes 9 core sample schools that replaced at least half of their teaching staff in 2010-11. 
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Exhibit B.12. 

Impetus for Increased Learning Time 


Summary 

This analysis examines whether SIG acted as the main impetus for increased learning time for 
the core sample schools implementing this improvement action in 2010-11 (as identified 
through the analysis of implemented improvement actions discussed in Exhibit B.8). See 
Chapter 5 for a discussion of the analysis, including analytic results. 


Technical Detail 

Data Sources 

• Interviews with district administrators, principals, teachers, and instructional coaches, 
including the following questions to elicit responses on increased learning time (Note 
that information may also have been obtained through other points in the interview, not 
just in direct response to the questions listed below.): 

District administrator 

o For each school, what instructional improvement strategies were 

planned/implemented this school year? What is the rationale behind these 
strategies? 

Principal 

o Could you describe the specific improvement strategies your school has 
implemented this school year? How have you prioritized these strategies? 

Teacher 

o Are there some broad approaches or strategies that the school as a whole is 

following to reach its improvement goals? What are they and do you think they are 
appropriate or likely to be effective? 

o Can you describe some of the specific improvement strategies that you know are, or 
will be, adopted at your school this school year as part of SIG? 

Instructional Coach 

o Are there some broad approaches or strategies that the school as a whole is 

following to reach its improvement goals? What are they and do you think they are 
appropriate or likely to be effective? 

o Can you describe some of the specific improvement strategies that are being, or will 
be, adopted this school year? 

• Focus groups with school improvement teams, teachers, parents, and students, including 
the following questions to elicit responses on increased learning time (Note that 
information may also have been obtained through other points in the focus group, not 
just in direct response to the questions listed below. Students were not explicitly asked 
questions specific to improvement actions.): 

School improvement team 

o How do you think the SIG improvement approaches/strategies identified for the 
school will address the issues facing the school? Please describe how you think 
specific improvement strategies are working, or will work, to improve the school. 

o How do you think the school improvement approaches and strategies are affecting 
your work at the school? 

Teacher 

o Can you describe some of the specific improvement strategies that were, or will be, 
adopted at your school this school year? 

o Do you think the improvement strategies fit the needs of the school and/or 
students? 

o What will be the greatest challenges to implementing these strategies? 

Parent 

o What kinds of activities to improve things for students at this school have you 
observed or heard about? 
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Exhibit B.12. 

Impetus for Increased Learning Time (continued from previous page) 



Technical Detail 

Stage 1: 
Qualitative 
Data Analysis 
Procedures 

Qualitative data were coded according to the codebook in Appendix A (see Chapter 2 for a 
description of the coding procedures). To examine the impetus for increased learning time, 
site visitors responded to the following question in the online data repository based on the 
coded data (specific codes used are provided in brackets): 

• Provide details on the strategies focused on use of time. In particular, if the school 
implemented an extended day strategy, please indicate how many minutes, and when 
the time was added. [$Use of time, *Change Strategy] 

After the first site visitor completed the data repository for his or her school(s), the second site 
visitor reviewed the responses to ensure the accuracy of the entries. 

Stage 2: 
School 
Classification 
Procedures 

Using the responses to the questions above and all coded data on strategies related to 
learning time for students [$Use of time], analysts identified information related to the 
impetus for increased learning time. Identification criteria are described below. For this 
analysis, "respondents" refers to individuals from the respondent groups listed above. When 
the analysis was complete, the site lead for each school reviewed and verified the results for 
his or her school(s). In cases of disagreement, the analysts and site lead returned to the coded 
data for the particular school(s) in question to resolve the disagreement. 

The threshold for classifications procedures using qualitative data may be perceived as low; 
however, this threshold was frequently exceeded. 

SIG identified as an impetus 

• At least one respondent reported that increased learning time began in 2010-11 to 
comply with SIG requirements. 

Other district reform identified as an impetus 

• At least one respondent reported that increased learning time was implemented in 
2010-11 or prior to comply with the requirements of a district reform effort. 

Notes 

Includes 20 of the 22 core sample schools that implemented increased learning time in 2010- 
11. Two schools were excluded from this analysis due to insufficient qualitative data (i.e., 
respondents did not explicitly address the topic). 
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Exhibit B.13. 

Increased Learning Time in Core Sample Schools 


Summary 

This analysis examines the implementation of increased learning time for the core sample 
schools implementing this improvement action in 2010-11 (as identified through the analysis 
of implemented improvement actions discussed in Exhibit B.8). See Chapter 5 for a discussion 
of the analysis, including analytic results. 


Technical Detail 

Data Sources 

• Interviews with district administrators, principals, teachers, and instructional coaches, 
including the following questions to elicit responses on increased learning time (Note 
that information may also have been obtained through other points in the interview, not 
just in direct response to the questions listed below.): 

District administrator 

o For each school, what instructional improvement strategies were 

planned/implemented this school year? What is the rationale behind these 
strategies? 

Principal 

o Could you describe the specific improvement strategies your school has 
implemented this school year? How have you prioritized these strategies? 

Teacher 

o Are there some broad approaches or strategies that the school as a whole is 

following to reach its improvement goals? What are they and do you think they are 
appropriate or likely to be effective? 

o Can you describe some of the specific improvement strategies that you know are, or 
will be, adopted at your school this school year as part of SIG? 

Instructional Coach 

o Are there some broad approaches or strategies that the school as a whole is 

following to reach its improvement goals? What are they and do you think they are 
appropriate or likely to be effective? 

o Can you describe some of the specific improvement strategies that are being, or will 
be, adopted this school year? 

• Focus groups with school improvement teams, teachers, parents, and students, including 
the following questions to elicit responses on increased learning time (Note that 
information may also have been obtained through other points in the focus group, not 
just in direct response to the questions listed below. Students were not explicitly asked 
questions specific to improvement actions.): 

School improvement team 

o How do you think the SIG improvement approaches/strategies identified for the 
school will address the issues facing the school? Please describe how you think 
specific improvement strategies are working, or will work, to improve the school. 

o How do you think the school improvement approaches and strategies are affecting 
your work at the school? 

Teacher 

o Can you describe some of the specific improvement strategies that were, or will be, 
adopted at your school this school year? 

o Do you think the improvement strategies fit the needs of the school and/or 
students? 

o What will be the greatest challenges to implementing these strategies? 

Parent 

o What kinds of activities to improve things for students at this school have you 
observed or heard about? 
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Exhibit B.13. 

Increased Learning Time in Core Sample Schools (continued from previous page) 


Stage 1: 
Qualitative 
Data Analysis 
Procedures 


Technical Detail 

Qualitative data were coded according to the codebook in Appendix A (see Chapter 2 for a 
description of the coding procedures). To examine increased learning time, site visitors 
responded to the following question in the online data repository based on the coded data 
(specific codes used are provided in brackets): 


• For each change strategy focused on the use of time listed below, please identify 
whether the following school respondents identified this strategy as being planned or 
implemented during the 2010-11 academic year (check as many as apply). [$Use of time, 
*Change Strategy] 

School 

District Impvt. 

Official Principal Teachers Team Coaches Parents 

Extended day 

Restructure day (e.g., block 
scheduling) 

Weekend school 
After-school or before- 
school tutoring 
Summer school 
Other (please specify) 


• Provide details on the strategies focused on use of time. In particular, if the school 


implemented an extended day strategy, please indicate how many minutes, and when 
the time was added. [$Use of time, *Change Strategy] 

After the first site visitor completed the data repository for his or her school(s), the second site 


Stage 2: 

Classification 

Procedures 


visitor reviewed the responses to ensure the accuracy of the entries. 

Using the responses to the questions above and all coded data on strategies related to 
learning time for students [$Use of time], for each action focused on the use of time, analysts 
identified whether the improvement action was being planned or implemented during the 
2010-11 academic year using the criteria described below. When the analysis was complete, 
the site lead for each school reviewed and verified the results for his or her school(s). In cases 
of disagreement, the analysts and site lead returned to the coded data for the particular 
school(s) in question to resolve the disagreement. For this analysis, "respondents" refers to 
individuals from the respondent groups listed above. 

The threshold for classifications procedures using qualitative data may be perceived as low; 
however, this threshold was exceeded in all schools. We determined one respondent was 
sufficient because implementation of increased learning time is a factual, unambiguous action 
that was not subject to a given respondent's judgment. 


Notes 


Identified as being implemented 

• At least one respondent identified the improvement action as being implemented. 
Includes 22 core sample schools that implemented increased learning time in 2010-11. 
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Exhibit B.14. 

Perceptions of Increased Learning Time 


Summary 

This analysis examines perceptions of increased learning time for the core sample schools that 
increased learning time in 2010-11 (as identified through the analysis of implemented 
improvement actions discussed in Exhibit B.8). See Chapter 5 for a discussion of the analysis, 
including analytic results. 


Technical Detail 

Data Sources 

• Interviews with district administrators, principals, and teachers, including the following 
questions to elicit responses on increased learning time (Note that information may also 
have been obtained through other points in the interview, not just in direct response to 
the questions listed below.): 

District administrator 

o For each school, what instructional improvement strategies were 

planned/implemented this school year? What is the rationale behind these 
strategies? 

Principal 

o Could you describe the specific improvement strategies your school has 
implemented this school year? How have you prioritized these strategies? 

Teacher 

o Are there some broad approaches or strategies that the school as a whole is 

following to reach its improvement goals? What are they and do you think they are 
appropriate or likely to be effective? 

o Can you describe some of the specific improvement strategies that you know are, or 
will be, adopted at your school this school year as part of SIG? 

Instructional Coach 

o Are there some broad approaches or strategies that the school as a whole is 

following to reach its improvement goals? What are they and do you think they are 
appropriate or likely to be effective? 

o Can you describe some of the specific improvement strategies that are being, or will 
be, adopted this school year? 

• Focus groups with school improvement teams, teachers, parents, and students, including 
the following questions to elicit responses on increased learning time (Note that 
information may also have been obtained through other points in the focus group, not 
just in direct response to the questions listed below. Students were not explicitly asked 
questions specific to improvement actions.): 

School improvement team 

o How do you think the SIG improvement approaches/strategies identified for the 
school will address the issues facing the school? Please describe how you think 
specific improvement strategies are working, or will work, to improve the school. 

o How do you think the school improvement approaches and strategies are affecting 
your work at the school? 

Teacher 

o Can you describe some of the specific improvement strategies that were, or will be, 
adopted at your school this school year? 

o Do you think the improvement strategies fit the needs of the school and/or 
students? 

o What will be the greatest challenges to implementing these strategies? 

Parent 

o What kinds of activities to improve things for students at this school have you 
observed or heard about? 
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Exhibit B.14. 

Perceptions of Increased Learning Time (continued from previous page) 



Technical Detail 

Stage 1: 
Qualitative 
Data Analysis 
Procedures 

To examine respondent perceptions of increased learning time, analysts reviewed all coded 
data associated with use of time and respondent support [$Use of time, *Change Strategy , 
%Buy-ln]. Qualitative data were coded according to the codebook in Appendix A (see Chapter 
2 for a description of the coding procedures). 

Based on the coded data, analysts identified information on respondent perceptions of 
increased learning time, noting how many and which types of respondents provided data. 

Stage 2: 
School 
Classification 
Procedures 

Using the coded data, analysts categorized schools using the classification scheme on 
perceptions of increased learning time described below. When the classifications were 
complete, the site lead for each school reviewed and verified the categorizations for his or her 
school(s). In cases of disagreement, the analysts and site lead returned to the coded data for 
the particular school(s) in question to resolve the disagreement. For this analysis, 
"respondents" refers to individuals from the respondent groups listed above. 

Positive 

• At least two respondents described the implementation of increased learning time as 
beneficial to the school (i.e., enriches learning opportunities for students, provides 
students with extra support); AND 

• No respondent described increased learning time as detrimental to the school (i.e., extra 
time not used effectively, staff not supported in implementing increased learning time). 

Mixed 

• Respondents disagreed about the implementation of increased learning time. 

Negative 

• At least two respondents described the implementation of increased learning time as 
detrimental to the school (i.e., extra time not used effectively, staff not supported in 
implementing increased learning time); AND 

• No respondent described increased learning time as beneficial to the school (i.e., 
enriches learning opportunities for students, provides students with extra support). 

Caveats 

This analysis is not an objective examination of the effectiveness of increased learning time 
but rather an aggregate reflection of respondents' perceptions regarding this improvement 
action. 

Notes 

Includes 10 of the 22 core sample schools that implemented increased learning time in 2010- 
11. Twelve schools were excluded from this analysis due to insufficient qualitative data (i.e., 
respondents did not explicitly address the topic). 
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Exhibit B.15. 

Student Behavior Reforms in Core Sample Schools 


Summary 

This analysis examines the implementation of student behavior reforms for the core sample 
schools that implemented student behavior programs or policies in 2010-11 (as identified 
through the analysis of implemented improvement actions discussed in Exhibit B.8). See 
Chapter 5 for a discussion of the analysis, including analytic results. 


Technical Detail 

Data Sources 

• Interviews with district administrators, principals, teachers, and instructional coaches, 
including the following questions to elicit responses on student behavior reforms (Note 
that information may also have been obtained through other points in the interview, not 
just in direct response to the questions listed below.): 

District administrator 

o For each school, what instructional improvement strategies were 

planned/implemented this school year? What is the rationale behind these 
strategies? 

Principal 

o Could you describe the specific improvement strategies your school has 
implemented this school year? How have you prioritized these strategies? 

Teacher 

o Are there some broad approaches or strategies that the school as a whole is 

following to reach its improvement goals? What are they, and do you think they are 
appropriate or likely to be effective? 

o Can you describe some of the specific improvement strategies that you know are, or 
will be, adopted at your school this school year as part of SIG? 

Instructional Coach 

o Are there some broad approaches or strategies that the school as a whole is 

following to reach its improvement goals? What are they, and do you think they are 
appropriate or likely to be effective? 

o Can you describe some of the specific improvement strategies that are being, or will 
be, adopted this school year? 

• Focus groups with school improvement teams, teachers, parents, and students, including 
the following questions to elicit responses on student behavior reforms (Note that 
information may also have been obtained through other points in the focus group, not 
just in direct response to the questions listed below. Students were not explicitly asked 
questions specific to improvement actions.): 

School improvement team 

o How do you think the SIG improvement approaches/strategies identified for the 
school will address the issues facing the school? Please describe how you think 
specific improvement strategies are working, or will work, to improve the school. 

o How do you think the school improvement approaches and strategies are affecting 
your work at the school? 

Teacher 

o Can you describe some of the specific improvement strategies that were, or will be, 
adopted at your school this school year? 

o Do you think the improvement strategies fit the needs of the school and/or 
students? 

o What will be the greatest challenges to implementing these strategies? 

Parent 

o What kinds of activities to improve things for students at this school have you 
observed or heard about? 
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Exhibit B.15. 

Student Behavior Reforms in Core Sample Schools (continued from previous page) 


Technical Detail 

Qualitative data were coded according to the codebook in Appendix A (see Chapter 2 for a 
description of the coding procedures). To examine student behavior reforms, site visitors 
responded to the following question in the online data repository based on the coded data 
(specific codes used are provided in brackets): 

• For each change strategy focused on school climate listed below, please identify whether 
the following respondents identified this strategy as being planned or implemented 
during the 2010-11 academic year (check as many as apply). [C_Attendance, C_behavior, 
C_Culture, *Change Strategy] 



• Among the improvement strategies being implemented this school year, did a subset of 
these stand out as the most salient strategies? In addition, please describe any features 
of the school change strategies on which you would like to elaborate [^Priorities, 

*Change Strategy]. 

After the first site visitor completed the data repository for his or her school(s), the second site 
visitor reviewed the responses to ensure the accuracy of the entries. 

Using the repository responses to the questions above, for each action focused on student 
behavior, analysts identified whether the improvement action was being planned or 
implemented during the 2010-11 academic year using the criteria described below. When the 
classifications were complete, the site lead for each school reviewed and verified the results 
for his or her school(s). In cases of disagreement, the analysts and site lead returned to the 
coded data for the particular school(s) in question to resolve the disagreement. For this 
analysis, "respondents" refers to individuals from the respondent groups listed above. 


Stage 2: 

Classification 

Procedures 


Stage 1: 
Qualitative 
Data Analysis 
Procedures 


Identified as being implemented 

• At least two respondents or the principal identified the improvement action as being 


Notes 


implemented. 

Includes 20 core sample schools that implemented school behavior reforms in 2010-11. 


B-36 


Case Studies of Schools Receiving School Improvement Grants 


Exhibit B.16. 

Perceptions of Student Behavior Programs and Policies 


Summary 

This analysis examines perceptions of student behavior programs and policies for the core 
sample schools that implemented student behavior programs and policies in 2010-11 (as 
identified through the analysis of implemented improvement actions discussed in Exhibit B.8). 
See Chapter 5 for a discussion of the analysis, including analytic results. 


Technical Detail 

Data Sources 

• Interviews with district administrators, principals, teachers, and instructional coaches, 
including the following questions to elicit responses on student behavior programs and 
policies (Note that information may also have been obtained through other points in the 
interview, not just in direct response to the questions listed below.): 

District administrator 

o For each school, what instructional improvement strategies were 

planned/implemented this school year? What is the rationale behind these 
strategies? 

Principal 

o Could you describe the specific improvement strategies your school has 
implemented this school year? How have you prioritized these strategies? 

Teacher 

o Are there some broad approaches or strategies that the school as a whole is 

following to reach its improvement goals? What are they, and do you think they are 
appropriate or likely to be effective? 

o Can you describe some of the specific improvement strategies that you know are, or 
will be, adopted at your school this school year as part of SIG? 

Instructional Coach 

o Are there some broad approaches or strategies that the school as a whole is 

following to reach its improvement goals? What are they, and do you think they are 
appropriate or likely to be effective? 

o Can you describe some of the specific improvement strategies that are being, or will 
be, adopted this school year? 

• Focus groups with school improvement teams, teachers, parents, and students, including 
the following questions to elicit responses on student behavior programs and policies 
(Note that information may also have been obtained through other points in the focus 
group, not just in direct response to the questions listed below. Students were not 
explicitly asked questions specific to improvement actions.): 

School improvement team 

o How do you think the SIG improvement approaches/strategies identified for the 
school will address the issues facing the school? Please describe how you think 
specific improvement strategies are working, or will work, to improve the school. 

o How do you think the school improvement approaches and strategies are affecting 
your work at the school? 

Teacher 

o Can you describe some of the specific improvement strategies that were, or will be, 
adopted at your school this school year? 

o Do you think the improvement strategies fit the needs of the school and/or 
students? 

o What will be the greatest challenges to implementing these strategies? 

Parent 

o What kinds of activities to improve things for students at this school have you 
observed or heard about? 
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Exhibit B.16. 

Perceptions of Student Behavior Programs and Policies (continued from previous page) 



Technical Detail 

Stage 1: 
Qualitative 
Data Analysis 
Procedures 

Qualitative data were coded according to the codebook in Appendix A (see Chapter 2 for a 
description of the coding procedures). To examine respondent perceptions of student 
behavior programs and policies, site visitors responded to the following questions in the online 
data repository based on the coded data (specific codes used are provided in brackets): 

• Among the improvement strategies being implemented this school year, did a subset of 
these stand out as the most salient strategies? In addition, please describe any features 
of the school change strategies on which you would like to elaborate [^Priorities, 

*Change Strategy]. 

• What are notable features of the school and its experience with SIG thus far? What 
distinguishes the change process thus far? In particular, please note any important 
relationships between the school context, individuals, strategies, and initial reports of 
progress or barriers to change. 

After the first site visitor completed the data repository for his or her school(s), the second site 
visitor reviewed the responses to ensure the accuracy of the entries. Using responses to the 
questions above, analysts identified data on respondent perceptions regarding student 
behavior programs and policies. 

Stage 2: 
School 
Classification 
Procedures 

Using the repository responses to the questions above, analysts categorized schools using the 
classification scheme on perceptions of student behavior programs and policies described 
below. When the classifications were complete, the site lead for each school reviewed and 
verified the categorizations for his or her school(s). In cases of disagreement, the analysts and 
site lead returned to the coded data for the particular school(s) in question to resolve the 
disagreement. For this analysis, "respondents" refers to individuals from the respondent 
groups listed above. 

Positive 

• At least two respondents described the implementation of student behavior programs 
and policies as effective in managing student behavior; AND 

• No respondent described student behavior programs and policies as ineffective in 
managing student behavior. 

Mixed 

• Respondents disagreed about the effectiveness of student behavior programs and 
policies. 

Negative 

• At least two respondents described the implementation of student behavior programs 
and policies as ineffective in managing student behavior; AND 

• No respondent described student behavior programs and policies as effective in 
managing student behavior. 

Caveats 

This analysis is not an objective examination of the effectiveness of student behavior programs 
and policies, but rather an aggregate reflection of respondents' perceptions regarding this 
improvement action. 

Notes 

Includes 15 of the 20 core sample schools that implemented school behavior programs and 
policies in 2010-11. Five schools were excluded from this analysis due to insufficient 
qualitative data. 
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Exhibit B.17. 

Visible Changes and Disruption From the Past 


Summary 

Case studies of the change process both in education and other industries describe the 
process of breaking traditional norms to interrupt complacency (Hassel & Hassel, 2009). A set 
of such visible changes may create a disruption from the past, a pivot point from which the 
turnaround process is demarcated. This analysis aims to identify whether study schools: (1) 
experienced visible changes; and (2) whether these changes together constitute a disruption 
from the past overall. See Chapter 6 for analysis results. 


Technical Detail 

Data Sources 

• Interviews with district administrators, principals, teachers, and instructional coaches 
(No specific questions were asked. Information for this analysis may have been captured 
at multiple points in the interview). 

• Focus groups with teachers (No specific questions were asked. Information for this 
analysis may have been captured at multiple points in the focus group). 

Stage 1: 

Identifying 

Indicators 

Because the literature on turning around low-performing schools does not include a 
comparable analysis on visible changes and disruptions from the past, the study team 
identified hypothesized indicators of a disruption from the past. The following eight 
indicators, or visible changes, are based either on specified elements of SIG (i.e., replacing the 
principal, replacing at least 50 percent of teachers, and experiencing a governance change) or 
were identified after preliminary analysis of interview data (i.e., implementing mandatory 
extension of the school day, changing school organization, changing the physical plant of the 
school, having public communication about school changes, and other visible changes). 
Implicit for each indicator is the assumption that such actions will create a visible interruption 
to school practices sufficient to disrupt business-as-usual. 

• Replacement of the principal. The school or district hired a new principal. 

• Replacement of at least 50 percent of teachers. At least 50 percent of teachers were new 
to the school at the start of the school year. 

• Governance change. An education management organization or charter management 
organization took over the school. 

• Mandatory extension of the school day. The school increased the length of the day, 
mandatory for all students. 

• Changes to school organization. This includes splitting into small learning communities, 
dividing the school into multiple smaller schools, combining with another school or 
school(s), or adding grades to the school. 

• Changes to the physical plant of the school. The school made physical updates to the 
school, such as rebuilding a school structure, repairing dysfunctional equipment (e.g. 
bathroom, cafeteria), or school-wide painting or aesthetic improvements. 

• Public communication about school changes. School changes were communicated to the 
public through such means as school media appearances (T.V., newspaper) and door-to- 
door campaigns aiming to disseminate information about the school. 

• Other visible changes. The school experienced other visible acts that do not fall into one 
of the above categories. For example, the principal of one school explained that he 
changed the name of the school to signal to the community that it was not the place it 
was before and that it was now a place with rigorous instruction and high expectations. 
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Exhibit B.17. 

Visible Changes and Disruption From the Past (continued from previous page) 



Technical Detail 

Stage 2: 
Qualitative 
Data Analysis 
Procedures 

Qualitative data were coded according to the codebook in Appendix A (see Chapter 2 for a 
description of the coding procedures). Based on the coded data, site visitors completed the 
following questions associated with the visible changes (specific codes used are provided for 
each question): 

• Was the principal new to the school in 2010-11 or 2009-10? [$Leadership] 

• Did the school replace at least 50 percent of teachers for 2010-11 or 2009-10? 
[$Staffing, *Change Strategy] 

• Was the school taken over by an EMO or CMO in 2010-11 or 2009-10? [$Governance, 
*Change Strategy] 

• Did the school implement mandatory extended day for all students in 2010-11 or 2009- 
10? [$Use of time, *Change Strategy] 

• Did the school change its organization, that is, did the school (a) split into multiple 
smaller schools, (b) combine with another school or school(s), (c) add grades to the 
building, or (d) split into small learning communities in 2010-11 or 2009-10? 
[$Governance, *Change Strategy] 

• Were there physical updates to the school of sufficient magnitude to constitute a 
departure from the past? Examples include (a) rebuilding a school structure, (b) 
repairing dysfunctional equipment (e.g., bathroom or cafeteria), or (c) school-wide 
painting or aesthetic improvement. Please present interview evidence to support your 
response. [C_Facilities] 

• Was there public communication about the school of sufficient magnitude to represent 
a departure from the past? Examples include (a) school media appearances (T.V., 
newspaper); and (b) door-to-door campaigns aiming to disseminate information about 
the school. Please present interview evidence to support your response. [C_Community 
relations] 

• Were there any other visible changes at the school of sufficient magnitude to represent 
a departure from the past? This is an opportunity to record information that was not 
capture by the other seven indicators. These changes may be symbolic or substantive. 
Please present interview evidence to support your response. 

Site visitors were then asked to respond to the following question, reflecting on the responses 
for the above questions: 

• Based on your review of qualitative coded data for your school, do you think the set of 
visible changes for your school together constitute a "disruption from the past" in 2010- 
11 and 2009-10? Yes or no? Please explain if needed. 

After the first site visitor completed the questions for his or her school(s), the second site 
visitor reviewed the responses to ensure the accuracy and completeness of the entries. 
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Exhibit B.17. 

Visible Changes and Disruption From the Past (continued from previous page) 



Technical Detail 

Stage 3: 

School 

Categorization 

Procedures 

For each indicator, two analysts first independently reviewed responses to the questions 
above to determine whether the action taken by the school was of sufficient magnitude 
(based on the preponderance of evidence from all respondent groups listed above) to 
represent a visible change from past years. In cases of disagreement, the analysts returned to 
the coded data for the particular school(s) in question to resolve the disagreement. 

Two analysts then independently classified the core sample schools on whether or not the set 
of visible changes as a whole constituted a disruption from the past, based on the definitions 
provided below. The use of four indicators as the threshold was informed primarily by a 
"natural" break in the data. As depicted in Exhibit 6.1, a cluster of 7 core sample schools 
experienced visible changes in at least four indicators, while the remaining 18 core sample 
schools experienced visible changes in two or fewer indicators. Independent coding by site 
visitors helped corroborate the decision to use four indicators as the threshold. For each 
school that experienced visible changes in at least four indicators, the site visitor coded the 
school as having experienced a disruption from the past in 2009-10 or 2010-11. 

When the classifications were complete, the site lead for each school reviewed and verified 
the categorizations for his or her school(s). In cases of disagreement, the analysts and site 
lead returned to the coded data for the particular school(s) in question to resolve the 
disagreement. 

Experienced a disruption from the past 

• The school demonstrated a visible change for at least four of the eight indicators 
(Indicators demonstrating a visible change are those which analysts determined were of 
sufficient magnitude based on the preponderance of evidence from all respondent 
groups listed above). 

Did not experience a disruption from the past 

• The school demonstrated a visible change for less than four of the eight indicators. 

Caveats 

As noted, this is an exploratory analysis for which the study team could not find a precedent 
in the existing literature. While we have attempted to ground our research in the SIG policy 
and existing literature, this measure is of unknown reliability. Moreover, the threshold 
established for a disruption (i.e., visible changes for at least four indicators) appears 
warranted by the data, but additional research is necessary to determine if it is appropriate. 

Notes 

Includes 25 core sample schools. 


B-41 


Case Studies of Schools Receiving School Improvement Grants 


Exhibit B.18. 

Centrality of SIG in the Change Process 


Summary 

This analysis focuses on the role of SIG as a perceived catalyst of the change process and how 
this perception is related to prior improvement efforts. See Chapter 6 for analysis results. 


Technical Detail 

Data Sources 

• Interviews with district administrators, principals, teachers, and instructional coaches, 
including the following questions to elicit responses on the centrality of SIG in the change 
process (Note that information may also have been obtained through other points in the 
interview, not just in direct response to the questions listed below. Instructional coaches 
were not explicitly asked questions specific to the centrality of SIG in the change process.): 

District administrator 

o Is there a districtwide approach to instruction improvement planned/being 

implemented for SIG schools? To what extent is the approach different from that of 
non-SIG schools, if at all? What specific strategies are involved in this approach? 
Please explain. 

o How are these instructional improvement strategies different from changes you've 
tried in the past, or from previous instructional approaches? How do priorities align 
with your previous reform efforts? 

Principal 

o How was [the SIG intervention model] selected as the SIG model to implement in this 
school? Who was involved in the decision? 

o Could you describe the specific improvement strategies your school has 
implemented this school year? How have you prioritized these strategies? 

Teacher 

o Can you describe some of the specific improvement strategies that you know are, or 
will be, adopted at your school this school year as part of SIG? 

• Focus groups with teachers, including the following questions to elicit responses on the 
centrality of SIG in the change process (Note that information may also have been 
obtained through other points in the focus group, not just in direct response to the 
questions listed below.): 

o Can you describe some of the specific improvement strategies that were, or will be, 
adopted at your school this school year? Do you know why your school is embarking 
on these strategies? 

Stage 1: 
Qualitative 
Data Analysis 
Procedures 

Qualitative data were coded according to the codebook in Appendix A (see Chapter 2 for a 
description of the coding procedures). To examine the relationship between the school's 
reform history and SIG, site visitors responded to the following question in the online data 
repository based on all coded data associated with school context [C_XX], in particular data that 
reference past reform efforts [C_Reform History]: 

• From the stakeholders' perspective, what are the defining features of the school context 
and history? These could pertain to the geographic setting, school facilities, student 
population, or special features of the school. Regarding the school's history, these could 
include changes in demographics, race relations, achievement levels, court decisions, 
reform history, etc. From which respondents did you draw these data? Please note the 
degree of consistency across stakeholders. [Context codes: C_XX] 
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Exhibit B.18. 

Centrality of SIG in the Change Process (continued from previous page) 



Technical Detail 

Stage 1: 
Qualitative 
Data Analysis 
Procedures 
(continued from 
previous page) 

• How would you characterize the relationship between the school's reform history and 
SIG? [Context codes: C_XX, C_Reform History] 

□ The school has engaged in few reform efforts in the past; SIG constitutes a major 
reform initiative. 

□ The school has engaged in limited reform efforts in the past, but SIG has mainly 
served to support existing practices. 

□ The school has started a turnaround or restructuring process prior to SIG, so SIG 
is serving to amplify or augment an on-going process. 

□ The school has a noteworthy history of reform churn, so SIG is one in a long 
series of reform efforts. 

□ The school's reform history is unclear. 

After the first site visitor completed the data repository for his or her school(s), the second site 
visitor reviewed the responses to ensure the accuracy and completeness of the entries. 

Using responses to the question above on the defining features of the school's history, analysts 
identified data that reflected the relationship between the school's reform history and SIG. This 
narrative could include statements such as the following: 

• "SIG coincided with a broader district reform initiative. In 2009-10, the district ranked all 
schools based on test scores and chose the lowest-performing (according to a district 
representative) 10 percent (according to a CMO representative) of schools. An external 
team conducted a qualitative assessment (described in the SIG application). The district 
assigned schools from this group to join the Hope Network in 2010-11, if they had not 
been already. Schools that were already in the Hope Network were assigned by the 
district to either become Voltaire Schools (an internal turnaround model) or Phoenix 
Academies (turned over to a CMO)." 

Stage 2: 
School 
Classification 
Procedures 

Using the coded data and school narratives, two analysts independently categorized schools 
based on the classification scheme on the centrality of SIG in the change process, described 
below. When the classifications were complete, the site lead for each school reviewed and 
verified the categorizations for his or her school(s). In cases of disagreement, the analysts and 
site lead returned to the coded data for the particular school(s) in question to resolve the 
disagreement. For this analysis, "respondents" refers to the principal and at least two other 
respondents from the respondent groups listed above. 

SIG catalyzed change that took place in 2010-11 

• Respondents described SIG funding and requirements as the primary impetus for change. 
That is, reform changes would not have happened were it not for the grant. For instance, 
SIG may require the replacement of a school's principal or staff (and then the new 
principal and/or new staff may drive change at the school). 
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Centrality of SIG in the Change Process (continued from previous page) 



Technical Detail 

Stage 2: 

School 
Classification 
Procedures 
(continued from 
previous page) 

SIG fit into an ongoing change process 

• Respondents described organizations or policies (e.g., district reforms) outside of SIG as 
the primary impetus for change, although the grant augmented or deepened ongoing 
reforms. For schools in this category, either a change process was launched prior to Year 1 
of SIG, OR a change process was planned prior to SIG and launched concurrent with Year 1 
of SIG. 

SIG supported business-as-usual 

• Respondents described 2010-11 changes as "marginal" or "tweaks," indicating that 
although the schools had made purchases with SIG funding, they had not launched a 
reform process. 

Caveats 

This analysis seeks to understand a school's reform history, but the quality of the data varied in 
accordance with the respondents' familiarity with each school's history. For example, in one 
school, turnover was such that only two staff members who were in the school in 2009-10 
were present and interviewed in 2010-11. 

Notes 

Includes 25 core sample schools. 
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School-Level Involvement in SIG Application Process 


Summary 

This analysis examines the involvement of school staff in the SIG application process. See 
Chapter 6 for a discussion of the analysis, including analytic results. 


Technical Detail 

Data Sources 

• Interviews with principals, teachers, and instructional coaches, including the following 
questions to elicit responses on the involvement of school staff in the SIG application 
process (Note that information may also have been obtained through other points in the 
interview, not just in direct response to the questions listed below. Instructional coaches 
were not explicitly asked questions specific to the SIG application process.): 

Principal 

o How was [the SIG intervention model] selected as the SIG model to implement in 
this school? Who was involved in the decision? 

Teacher 

o What has been your role in selecting, developing, or implementing any of the 
improvement strategies? Which strategies in particular? 

• Focus groups with school improvement teams and teachers (Note that information may 
also have been obtained through other points in the focus group, not just in direct 
response to the question listed below. Instructional coaches were not explicitly asked 
questions specific to the SIG application process.): 

Teacher 

o How did you find out about the reform efforts that are taking place this school year 
in your school? How were teachers kept informed? 

Stage 1: 
Qualitative 
Data Analysis 
Procedures 

Qualitative data were coded according to the codebook in Appendix A (see Chapter 2 for a 
description of the coding procedures). To examine the extent to which school-level 
respondents were involved in the SIG application process, site visitors responded to the 
following questions in the online data repository based on the coded data (specific codes 
used are provided in brackets): 

• Please describe the process through which this school received SIG funds - which may 
include issues related to the determination of SIG eligibility, the application process, and 
the award of funds. Who took the lead on the application? Were school-level 
stakeholders involved? If so, how limited or extensive was stakeholder involvement? 
[SIG_application process, SIGdistribution of funds] 

• Who was involved in the SIG application process? Check all that apply. [SIG_application 
process] 

□ District administrators 

□ Previous principal 

□ Current principal 

□ Teachers 

□ Other school leaders 

□ Parents 

□ Other community members 

□ Other (please specify) 

After the first site visitor completed the data repository for his or her school(s), the second 
site visitor reviewed the responses to ensure the accuracy and completeness of the entries. 
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Technical Detail 

Stage 2: 
School 
Classification 
Procedures 

Using the repository responses, analysts categorized schools based on the classification 
scheme on the level of involvement of school staff in the SIG application process, described 
below. Note that this analysis does not include a category for "significant involvement" as 
school-level respondents in none of the core sample schools described the principal and other 
school-level staff (i.e., school improvement team, instructional coaches) as having a 
significant role in the application process. When the classifications were complete, the site 
lead for each school reviewed and verified the categorizations for his or her school(s). In cases 
of disagreement, the analysts and site lead returned to the coded data for the particular 
school(s) in question to resolve the disagreement. For this analysis, "respondents" refers to 
the principal or at least the preponderance of evidence from all respondent groups listed 
above, though the data from the principal was given greater weight. 

Moderate school involvement 

• Respondents reported that the principal had a significant role in the application process, 
typically leading the application process or working in collaboration with district staff to 
inform the subgrant application; OR 

• Respondents reported that the principal had a limited role, but other individuals at the 
school, such as members of the school improvement team, instructional coaches, or 
parent representatives, in addition to the principal were also involved in the SIG 
application process. 

Limited school involvement 

• Respondents reported that the principal (but no other school staff) provided feedback or 
input to inform the subgrant application, but had no role in planning. 

No school involvement 

• Respondents either reported that they had no involvement in the application process 
(the district applied to receive SIG dollars and wrote the SIG application), or that they 
had been unaware of the SIG application process. 

Caveats 

This analysis relies on interview and focus group data from the end of the 2011 school year, 
which was approximately 10 months after the SIG application process would have occurred. 
Hence, the analysis could be compromised by respondent recall error. Additionally, principals 
who were new to the school during the first year of SIG may have been unaware of any 
involvement by their predecessors in the SIG application process. 

Notes 

Includes 25 core sample schools. 
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External Support for SIG Implementation From States, Districts, and 
External Providers 


Summary 

This analysis examines the sources of support (i.e., state, district, or external partners) for SIG 
implementation to core sample schools. See Chapter 6 for a discussion of the analysis, 
including analytic results. 

Technical Detail 

Data Sources 

• Interviews with principals, including the following questions to elicit responses on 
external support (Note that information may also have been obtained through other 
points in the interview, not just in direct response to the questions listed below.): 
o What services has your district provided to you this year as a SIG school? To what 
extent are these additional services (i.e., services non-SIG schools do not receive or 
services you did not receive last year)? 

o Who, other than the district and outside of the school staff (e.g., state staff, 

outside organizations, external commitments, etc.), provides support or assistance 
to your school? Please describe them and the type of support or assistance they 
provide. 

Stage 1: 
Qualitative 
Data Analysis 
Procedures 

To examine the sources from which core sample schools receive support for SIG 
implementation, analysts reviewed all coded data for the principal interviews associated with 
support provided by the state [*State and SIG_Support], the district [* District and 
SIG_Support], and the external support provider [*External support provider and 
SIG_Support]. Qualitative data were coded according to the codebook in Appendix A (see 
Chapter 2 for a description of the coding procedures). 

Using these data, analysts identified whether the principal for each school reported that their 
school received support from the state, district, and/or external support provider(s). 

Stage 2: 

Classification 

Procedures 

Using the coded data, for each source of support, analysts identified whether support was 
being provided using the criteria described below. When the analysis was complete, the site 
lead for each school reviewed and verified the results for his or her school(s). In cases of 
disagreement, the analysts and site lead returned to the coded data for the particular 
school(s) in question to resolve the disagreement. Because the principal was in the best 
position to describe external support, data from the principal constituted the minimum 
threshold for classification. 

Identified as a source of support 

• The principal reported that the school received support for SIG implementation from the 
source of support (i.e., state education agency, district, or external provider). 

Caveats 

Principal interviews served as the resource for determining if the school did or did not receive 
support for SIG implementation from the state, district, and/or external provider. 

Notes 

Includes 25 core sample schools. 
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Types of State and District Support for SIG Implementation 


Summary 

This analysis examines the types of support for SIG implementation received from states and 
districts. See Chapter 6 for a discussion of the analysis, including analytic results. 


Technical Detail 

Data Sources 

• Interviews with state administrators, district administrators, principals, teachers, and 
instructional coaches, including the following questions to elicit responses on external 
support (Note that information may also have been obtained through other points in the 
interview, not just in direct response to the questions listed below. Instructional coaches 
were not explicitly asked questions specific to external support): 

State Administrator 

o What is the state doing to support and improve chronically low-performing schools? 
How does SIG align with other state and federal school improvement efforts? 

o What technical assistance does your state provide specifically to districts and 
schools involved in SIG? 

o How does the statewide system of support integrate itself with or connect to SIG? 
Can you think of any examples of integration or duplication? 

District Administrator 

o How will the district support the instructional improvement strategies 
planned/implemented this school year in SIG schools? 

o If 1 were a principal in a SIG school in your district, can you tell me what type of 
support 1 would receive, whether from the state, the district, or another support 
provider? 

Principal 

o What services has your district provided to you this year as a SIG school? To what 
extent are these additional services (i.e., services non-SIG schools do not receive or 
services you did not receive last year)? 

o Who, other than the district and outside of the school staff (e.g., state staff, outside 
organizations, external commitments, etc.), provides support or assistance to your 
school? Please describe them and the type of support or assistance they provide. 

Teacher 

o What types of supports have you received to implement these improvement 
strategies? 

• Focus groups with teachers, including the following question to elicit responses on 
external support (Note that information may also have been obtained through other 
points in the focus group, not just in direct response to the question listed below.): 

o What types of supports have you received this school year to assist you in your 
teaching and to implement the reforms? 

Stage 1: 
Qualitative 
Data Analysis 
Procedures 

Qualitative data were coded according to the codebook in Appendix A (see Chapter 2 for a 
description of the coding procedures). To examine the types of supports for SIG 
implementation provided to schools, site visitors responded to the following questions in the 
online data repository based on the coded data (specific codes used are provided in brackets): 
• Did school or district stakeholders describe any support from the state education agency 
for SIG implementation? [*State, SIG_Support] 

□ Yes 

□ No 

□ Don't know/No data 
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Types of State and District Support for SIG Implementation (continued from previous page) 



Technical Detail 

Stage 1: 
Qualitative 
Data Analysis 
Procedures 
(continued from 
previous page) 

• If yes, please describe the support provided by the state education agency for SIG 
implementation. Which stakeholders described this support? [*State, SIG_Support] 

• Did school stakeholders describe any monitoring activities of SIG implementation by the 
state education agency? [SIG_Oversight] 

□ Yes 

□ No 

□ Don't know/No data 

• From the school stakeholders' perspective, please describe the district role in SIG 
implementation. Did the district support or impede SIG implementation? Please indicate 
which stakeholders commented on the district role. [SIGJmplementation, SIG_Support, 
SIG_Planning, SIG_Oversight, SIG distribution of funds and ^District] 

After the first site visitor completed the data repository for his or her school(s), the second 
site visitor reviewed the responses to ensure the accuracy and completeness of the entries. 
Using the responses to the questions above on respondents' perceptions of state and district 
support for SIG implementation, an analyst identified the following types of support, which 
were categorized into two broad categories: 

• Support for implementation of improvement actions 

o Providing support analyzing data, such as assisting in analyzing students' 
benchmark assessments during the school year, 
o Providing support and/or professional development in curriculum and instruction, 
including professional development for teachers who are expected to implement 
the improved curriculum or instructional strategies, 
o Facilitating a network of SIG schools, through which schools are provided a venue 
to obtain information and share promising practices, successes, and challenges, 
o Providing leadership support, such as assigning a leadership coach to the principal. 

• Compliance monitoring and guidance 

o Providing guidance on technical aspects of SIG requirements, such as what 
constituted extended learning time (e.g., whether it needed to be offered to all 
students or required for all students). 

o Monitoring SIG implementation, or providing feedback on whether SIG strategies 
were compliant with SIG federal guidance. 

To ensure that the data reflected only the perspective of school-level respondents (and not 
state or district administrators), analysts reviewed the coded data from school-level 
respondents' (i.e., principal, teachers, and instructional coaches) interviews and focus groups, 
and adjusted descriptions as necessary. In addition, analysts reviewed coded data from 
interviews with state and district administrators to determine if they also reported providing 
support to core sample schools, and if so, what types of support they provided. 
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Types of State and District Support for SIG Implementation (continued from previous page) 



Technical Detail 

Stage 2: 

Classification 

Procedures 

Using the principal reports identified in the repository responses, for each type of support, 
analysts identified whether support was being provided using the criteria described below. 
Because the principal was in the best position to describe external support, data from the 
principal constituted the minimum threshold for classification. Data from other school-level 
respondents, including teachers and instructional coaches, provided details about the types of 
support offered. When the analysis was complete, the site lead for each school reviewed and 
verified the results for his or her school(s). In cases of disagreement, the analysts and site lead 
returned to the coded data for the particular school(s) in question to resolve the 
disagreement. 

Identified as a type of support 

• The principal reported that the school received this type of support for SIG 
implementation. 

Notes 

Includes 25 core sample schools. 
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Perceptions of State and District Capacity to Support SIG Schools 


Summary 

This analysis examines the perceptions of state and district administrators on their capacity to 
support SIG schools. See Chapter 6 for a discussion of the analysis, including analytic results. 

Technical Detail 

Data Sources 

• Interviews with state and district administrators, including the following questions to 
elicit responses on support capacity (Note that information may also have been 
obtained through other points in the interview, not just in direct response to the 
questions listed below.): 

State administrator 

o 1 would like to know a little about how you perceive the current educational 
context (i.e., fiscal environment, political environment, etc.). Are there particular 
challenges you are facing now? How would you characterize your state's 
ability/capacity to meet these challenges? 
o How would you characterize the capacity of districts to implement SIG? How has 
district capacity changed since the implementation of SIG? 

District administrator 

o What support and guidance is your state department of education providing to 
your district to administer the School Improvement Grant? 
o If 1 were a principal in a SIG school in your district, can you tell me what type of 
support 1 would receive, whether from the state, the district, or another support 
provider? 

Stage 1: 
Qualitative 
Data Analysis 
Procedures 

To examine states' and districts' perceived capacity to support SIG schools, analysts reviewed 
the full state and district administrator interviews, with particular emphasis on coded data 
associated with state/district capacity [“"Capacity combined with *State/* District] . Both sets 
of interviews were analyzed to determine the extent to which respondents perceived that 
their own and each other's agency had the capacity to support SIG implementation. 
Qualitative data were coded according to the codebook in Appendix A (see Chapter 2 for a 
description of the coding procedures). 

Using these data, analysts identified data on perceptions of state and district capacity to 
support SIG implementation in SIG schools. 

Stage 2: 

Classification 

Procedures 

Using the coded data, analysts categorized states and districts based on the classification 
scheme on perceptions of capacity described below for both state capacity and district 
capacity separately. When the classifications were complete, the site lead for each school 
reviewed and verified the categorizations for his or her school(s). In cases of disagreement, 
the analysts and site lead returned to the coded data for the particular school(s) in question 
to resolve the disagreement. For this analysis, "respondents" refer to state and district 
administrators. Where discrepancies were noted between the state and district respondents, 
analysts deferred to self-reports— that is, state respondents for state capacity and district 
respondents for district capacity. 

Capacity to support SIG implementation 

• Respondents indicated that the state/district had the capacity to support SIG 
implementation. 

Limited or no capacity to support SIG implementation 

• Respondents indicated that the state/district had limited or no capacity to support SIG 
implementation. 
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Perceptions of State and District Capacity to Support SIG Schools 

(continued from previous page) 



Technical Detail 

Caveats 

This analysis is not based on an objective measure of state and district capacity (for example, 
taking into account financial resources, the number of SEA staff, etc.) but rather a reflection 
of state and district respondents' perceptions of their own and each other's agency's capacity 
to support SIG schools. 

Notes 

Includes 6 states and 13 districts in the core sample. 
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Perceived Improvement in Core Sample Schools 


Summary 

This analysis examines perceptions of improvement in the first year of SIG implementation in 
core sample schools. See Chapter 7 for a discussion of the analysis, including analytic results. 


Technical Detail 

Data Sources 

• Interviews with district administrators, principals, teachers, and instructional coaches, 
including the following question to elicit responses on perceived improvements (Note 
that information may also have been obtained through other points in the interview, not 
just in direct response to the question listed below. District administrators were not 
explicitly asked questions specific to perceived improvements.): 

Principal, teacher, and instructional coach 

o 1 know the SIG initiative is relatively new, but do you have any reflections on the 
strengths and weaknesses of the effort so far? To what extent are you satisfied 
with the progress made this school year? 

• Focus groups with teachers, students, and parents, including the following question to 
elicit responses on perceived improvements (Note that information may also have been 
obtained through other points in the focus group, not just in direct response to the 
question listed below. Students and parents were not explicitly asked questions specific 
to perceived improvements): 

Teacher 

o 1 know the SIG initiative is relatively new, but do you have any reflections on the 
strengths and weaknesses of the effort so far? To what extent are you satisfied 
with the progress made this school year? 

Stage 1: 

Identifying 

Indicators 

Research suggests that undertaking improvement strategies or actions does not necessarily 
translate into desired student outcomes, and achieving desired results can take some time 
(Aladjem et al., 2006). A set of school-level conditions appear to foreshadow student learning 
outcomes and could thus be thought of as intermediate outcomes or leading indicators of 
improvement. Based on a preliminary analysis of the interview and focus group data, the 
study team identified eight hypothesized leading indicators for which at least three 
respondents (from the respondent groups listed above) described improvement. All indicators 
were identified in the study's conceptual framework (see Chapter 1 for a discussion of the 
study's conceptual framework and leading indicators of improvement), with the exception of 
improving material resources, which was retained for this analysis because it was mentioned 
in 10 of the 25 core sample schools. 

• Teacher collaboration, often described in the literature as either same-subject teachers 
"identifying a common curriculum, and then analyzing common assessment data to 
make instructional changes" (DuFour, 2004b) or as teachers of the same students, but of 
different subjects, working together (Erb & Doda, 1989; Rottier, 2001). 

• Safe and orderly climate, an environment in which students "have a sense of being 
physically and psychologically safe in their school" (Consortium on Chicago School 
Research, 2004, Student-Centered Learning Climate section). 

• Quality of leadership, specifically the extent to which principals exhibit transformational 
leadership, instructional leadership, and strategic leadership. 

• Use of data for instructional decisions, characterized as the monitoring of student 
learning and frequent and transparent use of student outcome data to guide 
instructional decisions. 
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Technical Detail 

Stage 1: 
Identifying 
Indicators 
(continued from 
previous page) 

• Instructional practices, which consist of the interactions between teachers and students 
in the classroom around academic content. 

• Material resources, including the tangible supplies and supports available to enhance 
classroom practices and students' academic tasks, including technology, books, and 
manipulatives, but also more basic resources such as clean facilities with lockers and 
desks, as appropriate. 

• Student engagement, indicated by the extent to which teachers describe their students 
to be attentive and focused, contributing to academic tasks. 

• Staff expectations for students, specifically the short- and long-term objectives teachers 
express for their students, reflected in the rigor of the work they demand of students, 
and post-secondary activities. 

Stage 2: 
Qualitative 
Data Analysis 
Procedures 

To examine respondent perceptions of improvement in the areas listed above, analysts 
reviewed all coded data associated with each indicator (^Collaboration] for teacher 
collaboration; [C_Behavior, C_Safety] for safe and orderly climate; [*School leader, 
$Leadership] for quality of leadership; [$Data Use] for use of data for instructional decisions; 
[$lnstruction, $Curriculum] for instructional practices, [$Technology, $Curriculum] for 
material resources, [$Engagement] for student engagement, and [C_Commitment to 
students] for staff expectations for students). Qualitative data were coded according to the 
codebook in Appendix A (see Chapter 2 for a description of the coding procedures). 

Using these data, analysts identified data on perceptions of improvement for each indicator, 
noting how many and which types of respondents provided data. Examples of such evidence 
include: 

• "This was the first year where things were really being pulled together as far as how 
they collaborate with each other and being on par with the expectations for other 
schools in the district." (district administrator) 

• "1 have a daughter who dropped out of this school. It's a lot better now than it was. ...The 
teachers are much better this year than they have been, kids are working hard and staff 
is working hard. The kids are more interested in getting good grades now than just 
coming to school to socialize and slide by." (parent) 

• "We got a lot of new teachers last year from other areas, with expectations that they 
have been bringing from schools that are run how a school should be run - with 
expectations for students that you take finals, you turn in homework, and all of those 
things that you are supposed to do as a student, and the students are finally starting to 
get it. Now that we have new staff and new expectations, and they are actually more 
receptive to it than you would guess from a kid that went from not doing much to now 
all of a sudden you have to do your homework, you have to study for tests." (teacher) 

Stage 3: School 

Classification 

Procedures 

For each leading indicator, analysts first determined whether respondents perceived 
improvement. Indicators were classified as having made improvement if: 

• At least one of the following respondents— at least one teacher or the principal— and at 
least one respondent in one other respondent group (i.e., district administrator, 
instructional coach, parents, students) described improvement for the leading indicator, 
and no respondents disagreed or made statements to the contrary. For teacher 
collaboration, the threshold for perceived improvement was at least two teachers. 
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Perceived Improvement in Core Sample Schools (continued from previous page) 



Technical Detail 

Stage 3: School 
Classification 
Procedures 
(continued from 
previous page) 

Analysts then classified the core sample schools based on the classification scheme on 
perceived improvement described below. The use of three and six indicators as the thresholds 
was informed primarily by "natural" breaks in the data. As depicted in Exhibit 7.3, a cluster of 
6 core sample schools had reports of improvement for at least six indicators, while a cluster of 
8 schools had reports of improvement for three to five indicators and a cluster of 11 had 
reports of improvement for two or fewer indicators. When the classifications were complete, 
the site lead for each school was required to review and verify the categorization for his or 
her school(s). In cases of disagreement, the analysts and site lead returned to the coded data 
for the particular school(s) in question to resolve the disagreement. 

Reports of improvement in many areas 

• Perceived to have made improvement for at least six of the eight indicators. Schools 
were classified as having made improvement on a leading indicator if at least one of the 
following respondents— teachers or the principal— and at least one respondent in one 
other respondent group (i.e., district administrator, instructional coach, parents, 
students) described improvement for the indicator, and no respondents disagreed or 
made statements to the contrary. For teacher collaboration, the threshold for perceived 
improvement was at least two teachers; AND 

• Respondents described improvements in strong, illustrative language, such as a "light 
going on" or a "rebirth." 

Reports of improvement in some areas 

• Perceived to have made improvement for three to five of the eight indicators. 

Reports of improvement in few or no areas 

• Perceived to have made improvement for less than three of the eight indicators. 

Caveats 

This analysis is not a systematic examination of student outcomes and is not intended to 
imply causality with regard to SIG and student achievement. Rather this analysis is an 
aggregate reflection of respondents' perceptions of improvement for several leading 
indicators. Moreover, these measures do not capture the intensity of improvement within 
each indicator. 

Notes 

Includes 25 core sample schools. 
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Perceptions of Teacher Collaboration 


Summary 

This analysis examines perceptions of teacher collaboration. See Chapter 7 for a discussion of 
the analysis, including analytic results. 


Technical Detail 

Data Sources 

• Interviews with principals, teachers, and instructional coaches, including the following 
questions to elicit responses on teacher collaboration (Note that information may also 
have been obtained through other points in the interview, not just in direct response to 
the questions listed below. Instructional coaches were not explicitly asked questions 
specific to teacher collaboration.): 

Principal 

o How would you describe the staff in this school? 

Teacher 

o How would you describe the teaching staff at this school? What are the strengths and 
weaknesses as a staff? 

• Focus groups with teachers (Teachers were not explicitly asked questions specific to 
teacher collaboration; however, relevant information may have been mentioned at various 
points during the focus group.) 

• Teacher survey data 

Stage 1: 
Qualitative 
Data Analysis 
Procedures 

Qualitative data were coded according to the codebook in Appendix A (see Chapter 2 for a 
description of the coding procedures). To examine respondent perceptions of teacher 
collaboration, site visitors responded to the following question in the online data repository 
based on all coded data associated with collaboration ^Collaboration]. 

• Based on site visit and survey data, please describe teacher collaboration in this school. 
That is, how often do teachers meet to discuss classroom practices, curriculum, data, or 
challenges they are facing with students? The primary data sources for your response 
should be teacher interviews and focus groups, although the principal and instructional 
coach may provide data as well. ^Collaboration] 

Each site visitor reviewed the data in the repository for his or her school(s) to ensure the 
accuracy of the analyst's entries. 

Using responses to the above repository question, the analysts identified data that reflected the 
extent to which teachers collaborated at the school, noting how many and which types of 
respondents provided data. Examples of such evidence include: 

• "I'm more of the type of person who likes to work together. It doesn't work like that here. 
There are too many personalities and people stuck in their ways." ( teacher ) 

• "We've been able to plan lessons and use our grade-level time for looking at assessments 
and pinpoint areas based on that work together on who needs intervention. We've traded 
kids and had collaboration." ( teacher ) 

• "The principal does not fully realize how helpful the collaborative time is because there are 
times where she canceled it for things like the [teaching assistant] on the yard not doing a 
good job with the kids." (instructional coach) 

Stage 2: 
Teacher 
Survey Data 
Analysis 
Procedures 

Analysts incorporated data from three teacher survey items that measured teacher collaboration— 
Likert-scale items asking about the frequency of three collaborative activities: (1) consulting with 
other teachers about challenges faced in the classroom; (2) sharing lesson plans with other teachers; 
and (3) discussing the content of professional development activities with other teachers. 

Because individual survey items are less precise than survey scales, analysts only differentiated 
schools into two groups with the survey data— above the mean and below the mean— rather 
than three groups— 0.5 standard deviations below the scale mean or lower, within 0.5 standard 
deviations of the scale mean, and 0.5 standard deviations above the mean or higher— as was 
done for the survey scales. 
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Exhibit B.24. 

Perceptions of Teacher Collaboration (continued from previous page) 



Technical Detail 

Stage 3: 
School 
Classification 
Procedures 

Using the repository responses to the questions above and teacher survey data, analysts 
categorized schools based on the classification scheme on teacher collaboration described 
below. When the classifications were complete, the site lead for each school was required to 
review and verify the categorizations for his or her school(s). In cases of disagreement, the 
analysts and site lead returned to the coded data for the particular school(s) in question to 
resolve the disagreement. 

Culture of collaboration 

• Qualitative data: At least one of the following respondent groups— at least two teachers, 
the principal, or an instructional coach— reported that collaboration time is used for 
planning lessons, addressing individual student needs, or that time is otherwise described 
as productive; and at least two teachers described the working environment as collegial 
(i.e., "[teachers] are a team," "teachers not only work in grade level teams but that the 
school staff is like a family, we all work together"); AND 

• Survey data: School means on all three teacher collaboration items were above the overall 
sample means (3.63, 3.46, and 3.38). 

Some collaboration 

• Qualitative data: At least one of the following respondent groups— at least two teachers, 
the principal, or an instructional coach— reported that: (1) there is formal time allotted for 
collaboration during the school day, but at least one coach or one teacher suggested that 
the time was not used productively or that it was voluntary and not well-attended; or (2) 
there is informal collaboration (e.g., teachers sharing lesson plans), but no formal time 
during the school day that is used for collaboration; AND 

• Survey data: School mean on at least one of the three teacher collaboration items was 
below the overall sample mean (3.63, 3.46, and 3.38); OR 

• Qualitative data and teacher survey data did not match (e.g., all three survey items were 
above or below each of the overall sample means [3.63, 3.46, and 3.38], but qualitative 
data indicated a "culture of collaboration" or "inconsistent collaboration"). 

Inconsistent collaboration 

• Qualitative data: At least one of the following respondent groups— at least two teachers, 
the principal, or an instructional coach— reported that, while there is formal time during 
the school day allocated for collaboration, it is not used consistently for collaboration; OR 

• At least two teachers reported an absence of a culture of collaboration, through the 
following examples: teachers report feeling isolated, teachers report that leadership is not 
supportive of collaboration, or teachers are "off in their own rooms doing their own 
thing"; AND 

• Survey data: School means on all three teacher collaboration items were below the overall 
sample means (3.63, 3.46, and 3.38). 

Notes 

Includes 25 core sample schools. 
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Exhibit B.25. 

Perceptions of Safety and Orderliness of the School Environment 


Summary 

This analysis examines perceptions of the safety and orderliness of the core sample schools. 
See Chapter 7 for a discussion of the analysis, including analytic results. 

Technical Detail 

Data Sources 

• Interviews with district administrators, principals, teachers, and instructional coaches (No 
specific questions were asked. Information for this analysis may have been captured at 
multiple points in the interview.) 

• Focus groups with teachers, students, and parents (No specific questions were asked. 
Information for this analysis may have been captured at multiple points in the focus 
group.) 

Stage 1: 
Qualitative 
Data Analysis 
Procedures 

To examine respondent perceptions of school safety and orderliness, analysts reviewed all 
coded data associated with student behavior [C_Behavior] and safety of the school 
environment [C_Safety], Qualitative data were coded according to the codebook in Appendix 
A (see Chapter 2 for a description of the coding procedures). 

Using these data, analysts identified data on the school environment as reported by 
respondents, noting how many and which types of respondents provided data. 

Stage 2: School 

Classification 

Procedures 

Using the coded data, analysts categorized schools based on the classification scheme on 
perceptions of safety and orderliness described below. When the classifications were 
complete, the site lead for each school was required to review and verify the categorizations 
for his or her school(s). In cases of disagreement, the analysts and site lead returned to the 
coded data for the particular school(s) in question to resolve the disagreement. For this 
analysis, "respondents" refers to individuals from the respondent groups listed above. 
Safe/Orderly 

• At least two respondents explicitly described the school as safe or as having none or few 
behavior problems among students, and no respondents disagreed or made statements 
to the contrary. 

Mixed 

• Respondents made contradicting or differing statements about the school's safety and 
about student behavior. 

Unsafe/ Disorderly 

• At least two respondents described a feeling of being unsafe or behavior problems 
among students, and no respondents disagreed or made statements to the contrary. 

Caveats 

These school-level classifications do not include objective, quantitative indicators of student 
behavior or of incidents of crime. They are based only on the reported perceptions of 
interview and focus group participants. 

Notes 

Includes 25 core sample schools. 
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Exhibit B.26. 

Perceptions of the Use of Data for Instructional Decisions 


Summary 

This analysis examines perceptions of data use to inform instructional decisions. See Chapter 
7 for a discussion of the analysis, including analytic results. 


Technical Detail 

Data Sources 

• Interviews with district administrators, principals, teachers, and instructional coaches, 
including the following questions to elicit responses on the use of data (Note that 
relevant information may have been mentioned at various points during the interviews, 
not just in direct response to the questions listed below.): 

District administrator 

o For each school, what instructional improvement strategies were 

planned/implemented this school year? What is the rationale behind these 
strategies? 

Principal 

o Could you describe the specific improvement strategies your school implemented 
this school year? Have you prioritized these strategies? 

Teacher and instructional coach 

o Could you describe some of the specific improvement strategies that you know 
are, or will be, adopted at your school this school year as part of SIG? 

• Focus groups with teachers 

o Can you describe some of the specific improvement strategies that were, or will 
be, adopted at your school this school year? 

Stage 1: 
Qualitative 
Data Analysis 
Procedures 

To examine the extent to which schools were perceived as using data for instructional 
decisions, analysts reviewed all coded data associated with how data are being used by 
school administrators and staff [$Data Use]. Qualitative data were coded according to the 
codebook in Appendix A (see Chapter 2 for a description of the coding procedures). Using 
these data, analysts identified data on the frequency and purpose of data use as reported by 
respondents, noting how many and which types of respondents provided data. Examples of 
such evidence include: 

• "Academic interventions— Saturday school— been going on for past couple years, and 
you've had push in and push out. We've had that too. The difference is what material 
we are using. Before, students were targeted, and remediation was uniform. Now, 
remediation is much more targeted and based on data." (district administrator) 

• "Myself, I've done assemblies as 'here is our [state assessment] scores. This is where we 
are, and this is where we need to be. It's not okay to stay where we are.' And then 
providing kids tools as to what to do. If you're struggling, here are some online 
resources you can use at home. Here's how to interpret the [state standardized] test so 
you can diagnose yourself. We have links within our website too. That's brand new this 
year that we've started. With families, it's letters home and informational [sessions] 
during conferences." (principal) 

• "Outside of the class, we have academic coaches, so 1 am able to guide my instruction 
based on the data. We have common planning where we meet with reading coaches. 
We meet to plan our classes and look at data." (teacher) 
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Exhibit B.26. 

Perceptions of the Use of Data for Instructional Decisions (continued from previous page) 



Technical Detail 

Stage 1: 

Qualitative Data 
Analysis 
Procedures 
(continued from 
previous page) 

• "Data is a very strong weakness at our school. ..We have a data committee for the first 
time, but in terms of actual things they have produced, it hasn't been much. That's 
something we need to work on." (teacher) 

• "Most of the data analysis that has been done in the last three years has been with the 
academic coaches. After the reading assessments. ..I'll meet with grade-level team to 
discuss. After math benchmark assessment, the math coach will meet with them after. 
After the science benchmark, the science coach will meet with them [the teachers]. We 
also have a [Response to Intervention] coach that discusses data on individual students 
with Tier 2 and Tier 3 students." (reading instructional coach) 

Stage 2: School 

Classification 

Procedures 

Using the coded data, analysts categorized schools based on the classification scheme on 
perceptions of data use described below. When the classifications were complete, the site 
lead for each school was required to review and verify the categorizations for his or her 
school(s). In cases of disagreement, the analysts and site lead returned to the coded data for 
the particular school(s) in question to resolve the disagreement. For this analysis, 
"respondents" refers to individuals from the respondent groups listed above. 

High data use 

• At least three respondents described using data frequently and purposefully, such as to 
guide instruction (differentiating instruction) or to identify students to pull out of 
classrooms or for after-school instruction, or to guide professional development for 
teachers; AND 

• Respondents reported that the school had specific people who took responsibility for 
data use, such as instructional coaches, data teams, or whole-school processes. 

Medium data use 

• At least three respondents talked about using data to guide instruction or professional 
development for teachers. 

Low data use 

• Two or fewer respondents discussed reviewing student data; OR 

• Respondents specifically indicated that the school does not use student data. 

Caveats 

This is an aggregate reflection of respondents' perceptions of whether data are being used to 
guide instructional practices in the school. 

Notes 

Includes 25 core sample schools. 
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Exhibit B.27. 

Organizational Capacity in Core Sample Schools 


Summary 

This analysis examines the overall level of organizational capacity in core sample schools. See 
Chapter 7 for a discussion of the analysis, including analytic results. 


Technical Detail 

Data Sources 

• Interviews with district administrators, principals, teachers, and instructional coaches 

• Focus groups with school improvement teams, teachers, parents, and students 

• Teacher survey data 

Stage 1: 

Identifying 

Indicators 

Prior research on school improvement has identified a number of specific variables, or school 
conditions, often associated with schools with higher-than-expected student achievement, 
which together may be indicative of an overall level of organizational capacity. The study 
team identified the following eight indicators to gauge the overall school capacity of each 
core sample school (see Chapter 1 for a discussion of the study's conceptual framework and 
indicators of school capacity): 

• Leadership, specifically the extent to which principals exhibit transformational 
leadership, instructional leadership, and strategic leadership. 

• Coherence, or the degree to which the policies of a school reflect consistent goals; the 
strategies employed are clearly designed to foster achievement of these goals; and 
barriers and detractors from the goals and strategies are systematically removed (Honig 
& Hatch, 2004; Newmann et al., 2001). 

• Clear and shared goals, including a unity of purpose, explicit expectations, and shared 
values for student learning and success (Newmann et al., 2001; Purkey & Smith, 1983). 

• Teacher collaboration, often described in the literature as either same-subject teachers 
"identifying a common curriculum, and then analyzing common assessment data to 
make instructional changes" (DuFour, 2004b) or as teachers of the same students, but 
of different subjects, working together (Erb & Doda, 1989; Rottier, 2001). 

• Teacher-teacher trust, or the extent to which teachers feel they have mutual respect for 
each other, for those who lead school improvement efforts, and for those who are 
experts at their craft (Consortium on Chicago School Research, 2004). 

• Safe and orderly climate, an environment in which students "have a sense of being 
physically and psychologically safe in their school" (Consortium on Chicago School 
Research, 2004, Student-Centered Learning Climate section). 

• Use of data for instructional decisions, characterized as the monitoring of student 
learning and frequent and transparent use of student outcome data to guide 
instructional decisions (Coburn & Beuschel, 2012; Coburn & Taylor, 2012a; Coburn & 
Taylor, 2012b). 

• Locus of responsibility, characterized by the way in which school respondents attributed 
the performance problem in their school to factors within their control (i.e., internal 
causes) or outside of their control (i.e., external causes). 

Stage 2: 
School 
Classification 
Procedures 

Schools were categorized on each leading indicator. Classification schemes related to the 
locus of responsibility, teacher collaboration, safety and orderliness of the school 
environment, and use of data for instructional decisions, are presented in Exhibits B.4, B.18, 
B.19, and B.20. 
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Exhibit B.27. 

Organizational Capacity in Core Sample Schools (continued from previous page) 



Technical Detail 

Stage 2: 

School 
Classification 
Procedures 
(continued from 
previous page) 

For leadership, analysts aggregated the classifications for the three leadership dimensions— 
transformational leadership, instructional leadership, and strategic leadership (see Exhibits 
B.5, B.6, and B.7)— as follows: 

• High level. The principal was classified as "high on continuum" across all three 
leadership dimensions (transformational, instructional, and strategic). 

• Moderate level. The principal was not classified as "high on continuum" or "low on 
continuum" in more than two of the three leadership dimensions (transformational, 
instructional, and strategic). 

• Low level. The principal was classified as "low on continuum" across all three leadership 
dimensions (transformational, instructional, and strategic). 

Categorizations for coherence, clear and shared goals, and teacher-teacher trust were based 
exclusively on the teacher survey data. For clear and shared goals and teacher-teacher trust, 
analysts used the shared goals scale and the teacher trust scale (See Exhibit 2.7 for a detailed 
description of these scales). For coherence, analysts incorporated data from three teacher 
survey items that measured programmatic coherence— Likert-scale items asking about level 
of agreement with the following statements: "Once we start a new program, we follow up to 
make sure that it's working"; "1 worry that we are adopting too many different programs and 
practices in this school"; and "This school generally chooses only those school improvement 
activities that fit with our improvement goals and strategies." For each item, schools were 
assigned numeric values based on the school mean relative to the overall mean (l=at least 
0.5 standard deviations below the overall mean; 2=within 0.5 standard deviations of the 
overall mean; 3=at least 0.5 standard deviations above the overall mean; the second item 
was reverse-coded), which were summed to create a coherence index. Classification schemes 
related to these indicators of capacity are described below. 

Coherence 

o High level. Received a summative rating of at least 7 out of 9 on the coherence 
index. 

o Moderate level. Received a summative rating of 6 out of 9 on the coherence index. 

o Low level. Received a summative rating of less than 6 out of 9 on the coherence 
index. 

Clear and shared goals 

o High level. Shared goals scale average was at least 0.5 standard deviations (0.30) 
above the scale mean (3.18). 

o Moderate level. Shared goals scale average was within 0.5 standard deviations 
(0.30) of the scale mean (3.18). 

o Low level. Shared goals scale average was at least 0.5 standard deviations (0.30) 
below the scale mean (3.18). 

Teacher-teacher trust 

o High level of trust. Teacher trust scale average was at least 0.5 standard deviations 
(0.28) above the scale mean (2.93). 

o Average or moderate level of trust. Teacher trust scale average was within 0.5 
standard deviations (0.28) of the scale mean (2.93). 

o Low level of trust. Teacher trust scale average was at least 0.5 standard deviations 
(0.28) below the scale mean (2.93). 
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Exhibit B.27. 

Organizational Capacity in Core Sample Schools (continued from previous page) 



Technical Detail 

Stage 2: 

School 
Classification 
Procedures 
(continued from 
previous page) 

For each indicator, analysts assigned numeric values to the classifications (0 for the lowest 
category, 1 for the middle category, and 2 for the highest category), which were summed to 
create an aggregate index of school capacity. Analysts then classified the core sample schools 
based on the classification scheme on school capacity described below. Because there were 
no natural breaks in the distribution, cutpoints were set to divide the schools roughly into 
thirds. As depicted in Exhibit 7.6, a cluster of 8 schools had an overall capacity rating of 8 or 9, 
with 7 schools scoring a rating of 7 or below, and 6 schools scoring a rating of 10 or above. 
When the classifications were complete, the site lead for each school was required to review 
and verify the categorization for his or her school(s). 

Higher capacity 

• Received a summative rating of at least 10 out of 16 on the school capacity index. 
Moderate capacity 

• Received a summative rating of 8 or 9 out of 16 on the school capacity index. 

Lower capacity 

• Received a summative rating of less than 8 out of 16 on the school capacity index. 

Notes 

Includes 21 of 25 core sample schools. Four schools were excluded from this analysis because 
they were missing data on at least one of the eight indicators used to classify organizational 
capacity. In two schools, the leadership component of organizational capacity was based only 
on the measure of strategic leadership because these two schools were missing data on 
transformational and instructional leadership. 
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Appendix C. Analyses of Nonresponse Bias 


The teacher survey response rates varied across the 60 base sample schools, ranging from 0 percent in 
three schools to 100 percent in one school. The survey results for the 23 schools with less than a 50 
percent response rate were excluded from all analyses, due to concerns that the results for these 
schools lacked face validity. However, response rates in the low end of the 50 percent to 100 percent 
range still raise concerns of potential respondent bias and thus may not be representative of the 
attitudes and beliefs of teachers in these schools. The purpose of this appendix is to examine whether 
lower response rates in some schools may have introduced bias, and whether it is appropriate to report 
on data from schools with response rates that were only slightly above our 50-percent minimum 
threshold. 

Teacher rosters were provided by 92 percent (55 of 60) of the base sample schools. In the five schools 
where no rosters were provided, hard copies of the surveys were sent to schools to be filled out 
anonymously by teachers. Even in cases in which we could determine who did not respond to the 
teacher survey, we could not determine the characteristics of these nonrespondents. Given the absence 
of background data on nonrespondents, we conducted two other types of exploratory analyses based 
on the sample of respondents to investigate whether nonresponse might affect the survey results: 

1. Testing whether the school-level response rate predicted teachers' attitudes and beliefs, as 
measured by teachers' scale scores based on the survey (e.g., measures of principal leadership, 
principal trust, and school commitment). 

2. Testing whether the responses of later responders differed from the responses of earlier 
responders on the same scales. 

To carry out these two analyses, we pooled responses from teachers in the base sample schools, then 
estimated two-level hierarchical linear models (HLM) with teachers nested in schools, and adjusted for 
covariates. 41 Both analyses were conducted on each of the seven key survey scales, 42 omitting schools 
with less than a 50 percent response rate. 43 In both cases, scale scores were standardized to ease 
interpretation, using the overall teacher-level mean and standard deviation for teachers in sample 
schools with at least a 50 percent response rate. 

For the first set of analyses, the study team included the school-level response rate as an independent 
variable in the model, predicting teacher responses (see Exhibit C.l). If teachers in schools with lower 
response rates responded differently than teachers in schools with higher response rates, this might 
suggest that responding teachers differed from nonresponding teachers in their attitudes and beliefs. 
The results, however, indicate that the school response rate was not associated with respondents' 
attitudes/beliefs, as measured by any of the scales. For example, for schools with a response rate of 50 


41 The covariates in each model were based on the teacher survey and included the subject taught, a master's 
degree indicator, years of experience teaching, years of experience in their current school, the number of sections 
of students that the teacher taught, and the number of students enrolled in those sections. Missing data for any of 
these items excluded the teacher from the analysis. Approximately 10 percent of teachers were excluded from 
these analyses for this reason. Our results for each set of analyses were insensitive to the exclusion of all 
covariates from the statistical model. 

42 For items included in each scale and information on the reliability of these items, see Exhibit 2.7. 

43 We also ran these same models on schools with a response rate of 25 percent or greater, and our findings 
yielded similar results. Flowever, we felt that using schools with a response rate lower than 50 percent would not 
have face validity. 
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percent or higher, controlling for covariates, a 0.1 (or 10 percent) increase in school-level response rate 
was associated with an increase of 0.1*0.40=0.04 standard deviations in the principal instructional 
leadership scale score, which was not significant (p-value=0.62). 


Exhibit C.l. 

Relationship Between School-Level Response Rate and Survey Scales 



N of 

Teachers 

Standardized 

Coefficient 

p-value 

Principal Instructional Leadership 

1,064 

0.40 

0.62 

Principal Trust 

1,061 

0.07 

0.93 

School Commitment 

1,065 

0.13 

0.83 

School Resources 

1,063 

0.87 

0.19 

Shared Goals 

1,065 

-0.22 

0.75 

Shared Values 

1,065 

0.15 

0.78 

Student Behavior 

1,062 

0.21 

0.77 

Teacher Trust 

1,064 

0.26 

0.68 


Source: SST teacher survey, spring 2011. 

Notes: Includes 37 base sample schools (20 elementary, 17 high schools); 23 base sample schools were excluded from this 
analysis due to not meeting the 50 percent response rate threshold on the teacher survey. Covariates included in the model: 
subject taught, master's degree indicator, years of experience teaching, years of experience in their current school, number of 
class sections taught, number of student enrolled in their classes. 

*p-value < 0.1; **p-value < 0.05; ***p-value < 0.01 

For the second set of analyses, we included an indicator for teachers that responded late, defined as 
those who responded to the survey after the first four weeks of administration (see Exhibit C.2). 44 With 
the exception of the school resources scale, there was no statistically significant difference between late 
responders and early responders in their attitudes or beliefs. The result for the school resources scale 
indicates that within schools with a response rate of 50 percent or higher and controlling for covariates, 
teachers who took the survey after the first four weeks of administration reported that lack of school 
resources presented more of a challenge (p-value<0.05) than teachers who responded to the survey 
within four weeks of the administration. 


44 Because surveys were administered on a rolling basis, the survey launch date for each school and survey 
completion date were used to determine the length of time it took for teachers to respond. We conducted 
sensitivity analyses on each survey scale by defining a late responder as a teacher who responded after the first 
three weeks, and by defining a late responder as a teacher who responded after the first five weeks. In each 
analysis, the results were the same as those shown in Exhibit C.2, with one exception. When defining a late 
responder as a teacher who responded after the first five weeks, the analysis for the school resources scale ceases 
to be statistically significant (p-value=0.11). 
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Exhibit C.2. 

Difference Between Late Responders and Early Responders on Survey Scales 



N of 

Teachers 

Standardized 

Coefficient 

p-value 

Principal Instructional Leadership 

1,016 

-0.01 

0.86 

Principal Trust 

1,013 

-0.03 

0.68 

School Commitment 

1,017 

-0.01 

0.89 

School Resources 

1,015 

-0.18 

0.02** 

Shared Goals 

1,017 

-0.02 

0.80 

Shared Values 

1,017 

-0.05 

0.56 

Student Behavior 

1,014 

0.01 

0.86 

Teacher Trust 

1,016 

0.00 

1.00 


Source: SST teacher survey, spring 2011. 

Notes: Includes 37 base sample schools (20 elementary, 17 high schools); 23 base sample schools were excluded from this 
analysis due to not meeting the 50 percent response rate threshold on the teacher survey. Covariates included in the model: 
subject taught, master's degree indicator, years of experience teaching, years of experience in their current school, number of 
class sections taught, number of student enrolled in their classes. 

*p-value < 0.1; **p-value < 0.05; ***p-value < 0.01 

The results of these two analyses indicate that (1) there were no significant differences in the survey 
scales analyzed between teachers in schools with higher response rates and schools with lower 
response rates, and (2) there were virtually no significant differences in the survey scales analyzed 
between early and late responders. These analyses are not definitive, as in the absence of complete 
data about nonrespondents, we cannot determine with certainty whether the nonrespondents differed 
in systematic ways from the respondents. Thus, we cannot be absolutely certain that the observed 
school means would be the same if all teachers had responded in each school. Nevertheless, taken 
together, these results lend support to the hypothesis that nonrespondents and respondents did not 
differ substantially in their responses on the survey, and thus it may be reasonable to rely on the survey 
results to characterize schools with response rates above 50 percent. 
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Appendix D. Classifications Using Survey Data 


Many analyses reported in Chapters 3 and 7 were based on classifying schools with respect to features 
such as their context, reform activities, and practices. Several of these classifications were developed 
using multiple data sources: survey data, respondent interview and focus group data, and fiscal data. For 
classifications using survey data, either a survey scale or single item may have been used to help 
determine the overall classification of the school. The classification process for these analyses involved 
several steps. First, schools were classified separately based on survey data and on the qualitative data 
(interviews and focus groups). Then, the classifications were combined to form final classifications. 

When using survey scales to help classify schools, analysts first classified core sample schools into three 
categories: schools whose scale was 0.5 standard deviations below the scale mean or lower, schools 
whose scale was within 0.5 standard deviations of the scale mean, and schools whose scale was 0.5 
standard deviations above the mean or higher. 45 Where survey data could not be used due to low 
response rates (i.e., schools with less than 50 percent of teachers responding), classifications were 
determined by qualitative data alone. 46 

Classifications of core sample schools based on the survey data rely on a relative, rather than a criterion- 
based standard— that is, one comparing schools within the sample to each other versus one establishing 
an absolute threshold to distinguish between, for example, a "high" and "low" school. We decided on 
this approach for a few reasons. First, using the survey data as a criterion-based measure would require 
that we had an objective threshold to distinguish groups of schools - but no such threshold is known for 
the survey scales used in this study. 47 The schools in the core sample are not, and were not meant to be, 
a nationally-representative sample of schools, so we could not determine thresholds from our survey 
data alone. Second, by design, survey data facilitate relative assessments. The questions are asked in 
exactly the same way, and in the same order, for all respondents, facilitating valid comparisons. Finally, 
we determined that there was enough between-school variance (between 10 and 25 percent) on each 
survey scale to meaningfully distinguish amongst schools in our sample. Table D.l presents the survey 
scales and items used in each classification, as well as the relative cut points, using the threshold of 0.5 
standard deviations above and below the mean. 


45 Means and standard deviations were taken from the teacher-level data (rather than school-level averages), since 
the survey scales, from which this analysis is derived, measure teacher-level attitudes. Although we are ultimately 
classifying schools, these classifications are based on the average teacher in each school, so that our analysis is 
essentially comparing the average teacher in each school to the average teacher in our full sample of teachers. A 
0.5 standard deviation above and below the mean was set as the threshold as a way to ensure that the 'low' and 
'high' classifications were reasonably different from each other. That is, schools that are in the 'low' group have 
teacher respondents who, on average, responded at least one standard deviation lower than schools in the 'high' 
group. 

46 Response rates for each scale and survey item used in classifications were also required to be above 50 percent. 

47 It is possible to determine a prime facie reasonable threshold, such as "3" (which, in most survey items, 
corresponded to "agree"); however, even if the threshold seemed reasonable, it would still be arbitrary (e.g., why 
not '2.5'?). 
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Exhibit D.l. 

Teacher Survey Data Used in Classifications 



N of 

Teachers 

Mean 

Std. Dev. 

Mean - 0.5 Std. 
Dev. 

Mean + 0.5 Std. 
Dev. 

Survey Scales 






Principal Inst. Leadership 

690 

3.12 

0.66 

2.79 

3.45 

Principal Trust 

688 

3.09 

0.73 

2.73 

3.46 

School Commitment* 

691 

2.89 

0.69 

2.54 

3.23 

School Resources 

687 

2.63 

0.79 

2.23 

3.03 

Shared Goals 

691 

3.18 

0.60 

2.89 

3.48 

Shared Values* 

692 

3.13 

0.59 

2.84 

3.43 

Student Behavior* 

686 

2.03 

0.75 

1.65 

2.41 

Teacher Trust 

692 

2.93 

0.55 

2.65 

3.20 

Survey Items 






Program follow-up 

683 

2.72 




Adopting too many programs 

687 

2.28 




Programs fit with instructional goals 

682 

2.76 




Collective Efficacy 

685 

3.07 




Consult with other Ts about challenges 

679 

3.63 




Share the content of my lesson plans 

681 

3.46 




Discuss what I've learned from PD 

681 

3.38 





Source: SST teacher survey, spring 2011. 

Notes: Includes 21 of 25 core sample schools. Four schools were excluded for not meeting the 50 percent response rate 
threshold on the teacher survey (For these schools, classifications were based on qualitative data alone). 

Survey scales divided schools into three categories: schools whose scale was 0.5 standard deviations below the scale mean or 
lower, schools whose scale was within 0.5 standard deviations of the scale mean, and schools whose scale was 0.5 standard 
deviations above the mean or higher. Single survey items divided schools into two categories: schools at or above the mean, 
and schools below the mean. 

*Scale did not end up being used in school classifications. 


The following example illustrates in greater detail the ways in which qualitative and survey data were 
combined to create school-level classifications. For the analysis of perceived funding and resource 
constraints, analysts examined interview data for district administrators, principals, teachers, and 
instructional coaches, as well as focus group data for teachers (for more information, see Exhibit B.2). 
Using the qualitative data, analysts determined whether respondents, overall, believed that fiscal 
constraints outside of SIG posed a barrier to change, and grouped schools into three preliminary 
categories: 

1. Schools where non-SIG fiscal constraints were perceived as a barrier to school improvement, 

2. Schools where non-SIG fiscal constraints were perceived as a moderate challenge or 
perceptions were mixed, and 

3. Schools where non-SIG fiscal constraints were not perceived as a challenge. 

Analysts then separately grouped schools into three categories (low, medium, and high) using the school 
resources survey scale mean, based on whether each school's scale score was more than 0.5 standard 
deviations above the overall mean, within 0.5 standard deviations of the mean, or more than 0.5 
standard deviations below the mean. The qualitative data were then combined with the survey data to 
determine final classifications (see Exhibit D.2). 

The school resources scale was coded from 1 = major challenge to 4 = not a challenge. The school 
resource scale mean for core sample schools was 2.63, with a standard deviation of 0.79. For a school to 
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qualify for the "high" category (i.e., found school resources less of a challenge), the school mean would 
be 2.63 + (0.79/2) = 3.03 or higher. For a school to qualify for the "low" category (i.e., found school 
resources more of a challenge), the school mean would be 2.63 - (0.79/2) = 2.23 or lower. For a school 
to qualify for the "moderate" category, the school mean would have to fall between 2.23 and 3.03. For a 
school to have the final classification of having not perceived fiscal constraints outside of SIG to be a 
challenge, the qualitative rating had to suggest that non-SIG fiscal constraints were not a barrier, and 
the school resources scale would have to be 3.03 or higher. In this case, no schools met this threshold. 
For a school to have the final classification that the level of funding and resources perceived as a barrier 
to school improvement, the qualitative data had to suggest that non-SIG fiscal constraints were a 
barrier, and the school resources scale would have to be 2.23 or lower. 

For example, Coral High's qualitative data suggested that fiscal constraints outside of SIG did not pose a 
barrier to school change. However, survey data suggested that teachers found lack of school resources 
somewhat challenging. Therefore, the final classification for Coral High was that the school found fiscal 
constraints outside of SIG to be a moderate challenge or perceptions were mixed. 


Exhibit D.2. 

Classifications Example Using Qualitative Data and Survey Scales 




School 

Survey 



Qualitative Data 

Resources 

Data 


School 

Rating 

Scale * 1 2 3 

Rating 

Final Classification 

Coral High 

Not a challenge 

2.77 

Moderate 

Perceived as a moderate challenge or 
perceptions were mixed 

Inner Brooks High 

Moderate challenge 

2.60 

Moderate 

Perceived as a moderate challenge or 
perceptions were mixed 

Melon Elementary 

Moderate challenge 

3.25 

High 

Perceived as a moderate challenge or 
perceptions were mixed 

McAlliston High 

Moderate challenge 

n/a 

n/a 

Perceived as a moderate challenge or 
perceptions were mixed 

Big Acorn High 

Barrier 

2.50 

Moderate 

Perceived as a moderate challenge or 
perceptions were mixed 

Rossignol Elementary 

Barrier 

3.05 

High 

Perceived as a moderate challenge or 
perceptions were mixed 

Elmsville High 

Barrier 

2.22 

Low 

Perceived as a barrier to school improvement 

Peregrine Hill 
Elementary 

Barrier 

n/a 

n/a 

Perceived as a barrier to school improvement 


Source: SST respondent interview and focus groups, spring 2011; SST teacher survey, spring 2011. 

Notes: This table is for illustrative purposes only, and so contains only a select number of core sample schools. All school names 
are pseudonyms. 

a Survey data were used only for schools with a response rate of 50 percent or higher. For schools that did not meet the 50 
percent response rate threshold, classifications were based only on qualitative data. 

As individual survey items are less precise than survey scales, we used survey items differently than 
survey scales. In the case where survey item(s) were used to help classify schools, we grouped schools 
into two groups for each item - schools above the mean and schools below the mean. For example, for 
the analysis of teacher collaboration, analysts examined interviews with teachers, instructional coaches, 
and principals, as well as teacher focus groups (for more information, see Exhibit B.24). Using the 
qualitative data, analysts classified schools in three categories: 

1. Schools with a culture of collaboration, 

2. Schools with some collaboration, and 

3. Schools with inconsistent collaboration. 
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For each of the three survey items on teacher collaboration, analysts separately grouped schools into 
two categories (schools at or above the mean, and schools below the mean). The three teacher survey 
items that measured collaborative activities asked teachers "How often do you engage in the following 
activities": 

• Consult about challenges: Consult with other teachers about challenges I am facing in the 
classroom. 

• Share lesson plans: Share the content of my lesson plans with other teachers. 

• Discuss PD: Discuss what I've learned in professional development activities with other teachers. 

These items were coded from 1 = never to 4 = often. Although these items all measure aspects of 
teacher collaboration, they did not constitute a reliable scale (alpha = 0.64). Thus, analysts classified 
schools based on whether they were above or below the mean on each item. 

To create the final classification, the classifications based on the qualitative and survey data were 
combined. For a school to have the final classification of having a "culture of collaboration," the 
qualitative rating had to suggest a high level of collaboration, and all three survey items needed to be 
above the mean. For a school to have the final classification of "inconsistent collaboration," the 
qualitative data had to suggest low levels of collaboration, and all three survey items had to be below 
the mean (see Exhibit D.3). 

For example, Melon Elementary's qualitative data suggested that there is a high level of teacher 
collaboration at the school. Additionally, for each of the three survey items, the teachers on average in 
this school reported more frequent collaborative activities than teachers in the other core sample 
schools. Therefore, Melon Elementary was classified as having a culture of collaboration. 


Exhibit D.3. 

Classifications Example Using Qualitative Data and Survey Items 


School 

Qualitative 

Data 

Rating 

Survey Data: 
Consult T's 
about 
Challenges 

Survey Data: 
Share 

Lesson Plans 

Survey Data: 
Discuss PD 

Final Classification 



Mean: 3.63 

Mean: 3.46 

Mean: 3.38 


Melon Elementary 

High 

3.79 (above) 

3.79 (above) 

3.70 (above) 

Culture of collaboration 

Sawbuck Elementary 

High 

3.76 (above) 

3.59 (above) 

3.12 (below) 

Some collaboration 

Coral High 

Some 

3.61 (below) 

3.49 (above) 

3.56 (above) 

Some collaboration 

Rossignol Elementary 

Low 

3.55 (below) 

3.33 (below) 

3.62 (above) 

Some collaboration 

Big Acorn High 

Low 

3.51 (below) 

3.09 (below) 

3.16 (below) 

Inconsistent collaboration 


Source: SST respondent interview and focus groups, spring 2011; SST teacher survey, spring 2011. 

Notes: This table is for illustrative purposes only, and so contains only a select number of core sample schools. All school names 
are pseudonyms. 

a Survey data were used only for schools with a response rate of 50 percent or higher. For schools that did not meet the 50 
percent response rate threshold, classifications were based only on qualitative data. 
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Appendix E. Study of School Turnaround Budget Codebook 

Appendix E provides the codebook developed to analyze Year 1 (2010-11) SIG budgets. The codebook includes a list of all codes, along with a 
definition for each category. Examples of types of data that fell into each code category are listed, as were exclusions, when necessary. 

HUMAN CAPITAL MANAGEMENT 


Budget Line Item Purpose 

Definition 

Examples 

Exclusions 

Instructional coaches 

Staff to provide instructional support to teachers. 

Salaries, benefits, and/or stipends for newly- 
hired instructional coaches or staff serving 
part-time as instructional coaches. 


Instructional leadership 

Staff, services, or materials intended to support 
instructional leaders in the school. 

Salaries for new administrators; contracts with 
external organizations to provide leadership 
coaching. 


Professional development 

Staff, services, or materials for helping teachers 
acquire new skills, NOT including instructional 
coaches. 

Stipends for teachers to attend additional 
workshops or collaborate with each other; 
travel costs for teachers to attend conferences; 
contracts with external organizations to 
provide professional development; costs of 
substitutes to cover teacher release time. 


Teacher incentives 

Incentives paid to teachers for reaching specified 
goals or for recruitment or retention. 

Bonuses for recruitment or retention; award 
payments for reaching target test scores. 

Stipends for teachers for doing 
additional work (such as additional 
stipends for extended school days or 
additional training). 
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TECHNICAL CORE 


Budget Line Item Purpose 

Definition 

Examples 

Exclusions 

Academic student supports 

Staff, services, or materials that provide additional 
academic support to students, during or outside 
the school day. 

Salaries for additional classroom teachers, 
resource teachers, instructional aides; 
contracts with external organizations to 
provide tutoring; materials for tutoring 
programs or resource teachers. 

Software intended to help improve 
student's academic achievement 
(coded as curriculum and/or 
instructional changes and technology 
[hardware and software]). 

Curriculum and/or 
instructional changes 

Materials or technology supporting new content 
or pedagogical practices in classrooms. 

New textbooks; new supporting curricular and 
instructional materials; classroom technology 
such as SMART boards. 

Training for teachers on new 
curricula. 

Data use 

Staff, services, or materials that are intended to 
facilitate or increase data use among teachers. 

Salaries for data analysts; new data 
management software. 


Early childhood education 
programs 

Staff, services, or materials for programs for 
children younger than kindergarten-age. 

Salaries for preschool program staff; 
instructional materials for preschool programs. 


Extended day 

Staff, services, or materials for an extended school 
day, whether for all students or just a targeted 
group. 

Stipends for staff for longer work days; 
materials for after-school programs; contracts 
with external organizations to provide after- 
school programs; transportation for after- 
school programs. 


Extended week or year 

Staff, services, or materials for an extended school 
week or year, whether for all students or just a 
targeted group. 

Stipends for staff for Saturday school; salaries 
for teachers for summer school; instructional 
materials for summer bridge programs; 
contracts with external organizations to 
provide Saturday school programs; 
transportation costs. 


Nonacademic student 
supports 

Staff, services, or materials to support students' 
non-academic needs. 

Salaries for social workers; contracts with 
external organizations to provide student 
employment or to provide character 
education; character education software. 


Technology (hardware and 
software) 

Technology for any purpose (also coded in any 
other relevant category). 

SMART boards; software; parent 
communication software. 
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CONDITIONS THAT SUPPPORT TEACHING AND LEARNING 


Budget Line Item Purpose 

Definition 

Examples 

Exclusions 

Parent activities 

Staff, services, or materials to increase parent 
engagement or provide services to parents. 

Salaries for parent liaisons; stipends for 
teachers to attend training on parent 
engagement; technology to better 
communicate with parents; computers for 
English classes for parents. 


Strategies to change student 
behavior and/or increase 
school safety 

Staff, services, or materials that are intended to 
help manage student behavior or improve school 
safety. 

Salaries for security guards; contracts with 
external organizations to conduct training for 
teachers in bullying prevention. 



INDIRECT COSTS 


Budget Line Item Purpose 

Definition 

Examples 

Exclusions 

Indirect costs 

Unspecified overhead costs 
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